How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 47 of 67

The Scoring Made The Steps

By Joshua Bolick, Awareness Software Group ~14 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. These models do not learn a task all in one go. Different parts of it arrive at different moments, and an earlier result of ours measured how separate those moments are: on a simple task the parts arrive as distinct steps, while on a harder one they blur into a single long ramp. Everything about that measurement rested on a score that marks each prediction right or wrong, and a later result of ours showed that kind of score has a specific side effect. A part of the task cannot start scoring better until its right answer overtakes its nearest rival, which can delay it well past the point where the model actually started improving on it.

What we found. So we measured the same runs a second way, using how surprised the model is by the correct answer, which has no right-or-wrong line in it. We expected the steps to become clearer. They became about half as separated, on both tasks. The scoring was not blurring distinct stages together; it was pulling them apart. Underneath, the model improves on all the parts of the task far more evenly than the scoring suggests.

Why it matters. Two things are worth separating carefully here. The order in which the parts arrive is completely unaffected and remains highly repeatable from run to run, so the model really does work through the task in a consistent sequence. What changes is how *separately* it does so. We have added a dated note to the earlier page saying exactly this, rather than quietly editing it. And this is now the second time we have measured how much our scoring shapes a result: it turns out to matter very little for the overall picture of a model improving, and a great deal for the finer structure inside it. That is a more useful thing to know than either measurement on its own, and we only have it because both were run.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 10 training runs, no GPU, no cost.

Program v2 Bucket N, item N11. Decisive computation: analysis/unthresholded_staircase.py. Output: analysis/unthresholded_staircase.json. Reproduce with python analysis/unthresholded_staircase.py, or re-derive every endpoint with --reuse.

The question: two results that were never designed to meet

A12 disaggregated the transition by token class and applied a scale-free test, a stagger between classes counts only if it is large relative to how long a single class takes to transition. On delayed-copy by position that ratio is 1.442, a staircase. On dispatch-copy-8lag by announced lag it is 0.417: the acquisition order is just as reproducible (+0.910), but each class takes so long that the classes overlap, so it reads as a ramp.

**How separately a model learns its sub-skills, scored two ways.** These models do not learn a task all at once: different parts of it arrive at different moments. An earlier result of ours measured how separate those moments are, and found that on the first task they arrive as distinct steps while on the second they blur into a ramp. That measurement used a score that marks each prediction right or wrong. Here the same runs are measured a second way, using how surprised the model is by the correct answer, which has no right-or-wrong line in it. Without the right-or-wrong line the sub-skills are about half as separated, on both tasks. We expected the opposite: that the scoring was blurring distinct steps together. In fact the scoring is what pulls them apart, because a part of the task cannot start scoring better until its right answer overtakes its nearest rival, and that happens later for the harder parts. Underneath, the model improves on all the parts far more evenly than the scoring suggests. The order in which they arrive is unaffected and remains highly repeatable.

M6 then showed held-out accuracy is an argmax, that the same runs measured in bits per token keep 93% of the transition's sharpness, and that thresholding measurably sharpens a curve. An argmax does not only shift a curve: it widens it, because a class's accuracy cannot move until its correct token overtakes its rival.

So A12's denominator, the quantity that decided "ramp" rather than "staircase", is a thresholded quantity, and nobody had measured it without the threshold.

Kill test, fixed in advance: the stagger-over-width ratio measured in bits is not materially larger than the ratio measured in accuracy.

Design, and the flaw the smoke test caught

A12's runs, configuration, seeds and class schemes exactly, with one addition: every class's bits per token is recorded beside its accuracy at every evaluation, from the same forward pass. Both views then get the same shape analysis, made direction-aware so "width" means the same thing for a rising curve and a falling one.

Both views must be computed over the same classes, and the first version was not. A12's eligibility bar, a class must gain 0.3 before its shape is read: is in accuracy units, and bits fall by several units over a run, so applied naively the bits view admits classes the accuracy view rejects. The reduced-scale run showed exactly that: on dispatch-copy-8lag the accuracy view had no eligible classes while the bits view had several, which would have compared two different sets of curves and reported the difference as a threshold effect. Eligibility is now decided once, from the accuracy view, with both ratios computed over that same set, which also makes the per-run pairing exact.

Result: the kill test fires, and the direction is the opposite of the hypothesis

task	view	stagger	width	stagger / width
`delayed-copy`	accuracy	`75.0`	`52.0`	`1.442`
	bits	`48.0`	`68.0`	`0.706`
`dispatch-copy-8lag`	accuracy	`105.0`	`251.5`	`0.417`
	bits	`34.0`	`195.0`	`0.174`

task	paired difference, bits minus accuracy
`delayed-copy`	`−0.747` `[−0.957, −0.537]`
`dispatch-copy-8lag`	`−0.247` `[−0.304, −0.190]`

Both anchors reproduced exactly, 1.442 and 0.417, digit for digit with A12's published values, on A12's seeds and configuration.

Removing the threshold roughly halves the class separation on both tasks, and both paired intervals exclude zero. The argmax does not smear classes together; it pulls them apart.

What actually moves

The ratio has two components and they behave differently:

The stagger collapses. 75.0 → 48.0 on delayed-copy, and 105.0 → 34.0 on the dispatch task, a threefold fall. In bits, the classes begin improving at much more nearly the same time than their accuracy curves suggest.
The width moves either way. It grows on delayed-copy (52.0 → 68.0) and shrinks on dispatch (251.5 → 195.0), so the ratio's fall is driven by the numerator.

The reading that fits: the model's predictive distribution improves on all classes broadly together, and the argmax crossing happens at very different times per class because each class's correct token has to overtake its rival, which takes longer for the classes that are harder. **Much of what A12 measured as sequential acquisition is sequential threshold crossing.**

What this does and does not change about A12

A12's ordering result is untouched. Which classes resolve first is +0.983 consistent across seeds on delayed-copy and +0.910 on the dispatch task, and nothing here bears on that. The model really does acquire its sub-skills in a reproducible order.

A12's separation result is threshold-dependent, and by about a factor of two. "One jump is really four" is a statement about when each element's answer wins the argmax. Measured without the threshold the four are much closer together: still ordered, much less separated.

And the dispatch task's ramp is more of a ramp, not less. 0.174 in bits against 0.417 in accuracy. The kill test's motivating hypothesis: that thresholding was hiding a staircase there, is refuted in the strongest available direction.

A dated note has been appended to A12's record pointing here.

This is the third measurement of the same thing, and they agree in size order

what was measured	how much the threshold contributes
M6: sharpness of the overall curve	`8%` (`+0.336` `[+0.082, +0.591]` on `4.50`)
N11: separation between classes	about half

Thresholding matters little for the shape of the aggregate curve and a great deal for the structure between sub-skills. That is a more useful statement than either measurement alone, and it is available only because both were run.

Verdict

The kill test fires on both tasks, with paired intervals excluding zero.
The direction is opposite to the hypothesis: the threshold creates separation rather than hiding it.
Both anchors reproduced exactly.
A12's ordering stands; A12's separation is roughly halved without the threshold.
The premise check on N3 caught a duplicate, and re-reading A12 against M6 is what produced this item.

Limits

Five seeds per task, two tasks, one width, one learning rate.
"Width" is the 25%-to-75% crossing span, which is one of several defensible definitions and is A12's. A different definition would move both views, though not obviously by the same amount.
Bits and accuracy have different natural scales, so the eligibility bar cannot mean the same thing in both. That is handled by deciding eligibility once from accuracy, which makes the class sets identical but leaves the bar itself accuracy-shaped.
This does not identify what the classes are doing differently. It says their distributional improvement is more synchronous than their argmax crossings; it does not say why the crossings are ordered as they are, which is A12's rho +0.910 result and still unexplained mechanistically.
Nothing here bears on whether the aggregate transition is real. M6 settled that separately, in the other direction.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
argmax: Short for 'the argument that maximises'. A model outputs a score for every possible next token; the argmax is simply whichever one scored highest. Accuracy is measured on the argmax, which is why accuracy is a thresholded quantity: only the winner counts, and by how much it won is discarded.
bits per token: How surprised the model is by the correct answer, in the units used by compression. It is the training loss written a different way. Unlike accuracy it has no right-or-wrong threshold in it, so a gradual change in the model shows up as a gradual change in the number. Pure guessing on a 32-symbol vocabulary costs 5 bits per token.
held-out: Data the model was never trained on, kept back specifically to test it. Scoring a model on data it has already seen measures memorisation, not learning.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
reproducible: Someone else running the same code on the same inputs gets the same numbers. Every result here is checked this way.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.