How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

The Prerequisite Forecasts the Skill

By Joshua Bolick, Awareness Software Group ~11 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually happens at the moment a model learns? – It builds machinery rather than selecting it, working through the task in a reproducible order and trying a simpler wrong rule on the way. The visible training curve cannot tell you which is happening.

In plain English

What we asked. A 2026 paper showed that in language models, when one internal mechanism forms predicts when a more complex one will form, at a fixed multiple of the time. Our experiments have the same shape built in: combined skills and the simpler skills they are built from. So we tested the same rule on runs we already had, without training anything new.

What we found. It worked. Forecasting a combined skill as a fixed multiple of when its last building block was learned cut the error to about a third of a forecast that ignores the building blocks, and the multiple stayed almost the same across ten quite different setups. It is better at predicting typical timing than at ranking individual runs.

Why it matters. This is our new habit of reading recent research paying off directly: a rule from someone else's work, checked against our own data in an afternoon.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. No training: the committed runs of six earlier records. The script and kill test were committed before it was run.

Program v2 Bucket T, item T5, from the second standing source search (see docs/sources.md). Decisive computation: analysis/anchor_forecast.py. Output: analysis/anchor_forecast.json. Reproduce with python analysis/anchor_forecast.py (seconds).

The question

arXiv 2609.19000, "Capability Emergence Can Be Forecast: Per-Seed, In Advance, With Calibrated Intervals, Certified False Alarms, and a Blind Pre-Registered Gate" (Howe), forecasts when an induction head forms from when its mechanistic precursor, the previous-token head, forms, with a multiplicative rule (t_event ~ 1.19 x t_anchor) that passed blind validation on unseen configurations; its reported rank correlation across 30 same-configuration transformers is 0.977. Only the abstract was read.

**When a skill's building blocks are learned forecasts when the skill itself will be.** Three combined skills, each built from simpler ones, measured in ten different setups of earlier experiments. Each setup is predicted using only the others. Blue: forecast the combined skill as a fixed multiple of when its last building block was learned. Grey: forecast it without looking at the building blocks. *Watching the building blocks cuts the forecast error to a third or less, and the multiple barely changes between setups. A 2026 paper found the same kind of rule for a mechanism in language models.*

This programme has the same structure planted by design: composite skills (C = A + B, F = B + D, E = C + D) and the prerequisites they are built from, acquired per seed in six committed experiments -- O11, O12, O17, O20, O21 and O24 -- spanning five-, six- and seven-skill tasks, widths 64 and 128, two budgets and three teaching doses. Ten configurations qualify: every committed arm in which neither of a composite's prerequisites was manipulated.

The rule, fixed before execution: the anchor is the latest prerequisite's acquisition step; the forecast is k x anchor, with k the median ratio over the other configurations (each configuration held out in turn). The baseline: the median acquisition step of the composite over the other configurations, ignoring the anchor. Endpoint: mean over held-out configurations of the median absolute percentage error per run.

Kill test, fixed before execution: for C, the anchor rule's error is not smaller than the baseline's.

Result: the kill test does not fire

Composite	Configurations	Anchor rule error	Baseline error	`k` (pooled)	Mean within-configuration rank correlation
`C = A + B`	`10`	`16.6%`	`47.4%`	`2.21`	`0.47`
`E = C + D`	`10`	`10.2%`	`45.4%`	`1.61`	`0.45`
`F = B + D`	`9`	`26.0%`	`53.1%`	`3.22`	`-0.01`

Knowing when a composite's last prerequisite was acquired forecasts when the composite will be, blind, with a third of the baseline's error or less -- for every composite. The ratio is strikingly stable: across all ten held-out fits k for C stays between 2.15 and 2.29, although the configurations differ in task size, width, budget and how often an unneeded skill is taught. That is the same kind of stability 2609.19000 reports for its induction-head rule.

Where it is weaker than the paper: ranking seeds within one configuration. The paper reports 0.977 across same-configuration runs; here C and E average 0.45-0.47 and F about zero. Most of the rule's power is in carrying each configuration's scale -- a wider or less-crowded model learns everything faster, and the anchor measures that -- rather than in ordering individual seeds.

It is a forecast in advance. The anchor arrives when the composite is still k - 1 times further away: for C, about 1.2 anchor-lengths ahead.

What stands

Kill test does not fire. The anchor rule's blind error is 16.6% for C, against 47.4% for a forecast that ignores it; 10.2% for E and 26.0% for F.
The composite-to-prerequisite ratio is nearly constant across ten configurations (C: 2.15-2.29).
Per-seed ranking within a configuration is weak (0.47 for C, about zero for F), unlike the paper's induction heads.
This is the first positive forecasting result from the planted-skill tasks, and it is free: it uses acquisition times the runs already produce.

Limits

Abstract only for 2609.19000; its method is richer (calibrated intervals, certified false alarms) than the multiplicative rule tested here.
Configurations share one task family and one learning rate; the rule's constant is a property of this task, not a universal.
Acquisition is P3's threshold definition, the same for anchor and target; a different definition could change k.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

baseline: The thing you compare against. A result without one is not a result.
calibrated: A model is calibrated when its confidence matches how often it is right: answers it gives with 90% confidence should be right about 90% of the time.
held-out: Data the model was never trained on, kept back specifically to test it. Scoring a model on data it has already seen measures memorisation, not learning.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
transformer: The architecture behind most modern large language models. It uses attention to look at every part of the input at once, rather than reading in order.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

In plain English

The question

Result: the kill test does not fire

What stands

Limits

Terms on this page

Read next

Get new results as we publish them

Want this measured on your data?