How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Which Side of the Moment

By Joshua Bolick, Awareness Software Group ~13 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Part of a bigger question: How do we know our own results are real? – Repeatedly, the control rather than the measurement decided the result -- and several striking findings dissolved when the right comparison was finally run.

In plain English

What we asked. We had a training shortcut: partway through, switch to a cheaper setting that processes less data per step. On smaller models it reached the target on 22% less data. Repeated on a model four times wider with a harder task, it cost 26% more instead. Something in that change had flipped it, and three things had changed at once: the model got bigger, the task got harder, and each cheap step carried less data to learn from. So we ran a grid that varied them separately.

What we found. None of the three was responsible. What decides it is whether the switch happens before or after the moment the model suddenly improves. Every switch made before that moment saved about a fifth of the data. Every switch made after it cost between nine and forty-two per cent. One combination proves it on its own: the same model, the same task, the same data per step, switched at two different times, saving in one case and costing in the other. Model size does shift how good the outcome is, but it never flips it, and the amount of data per step decides nothing at all.

Why it matters. The uncomfortable part is what this says about the original result. We checked when the sudden improvement happened in those old runs: step 88. The signal we were using to decide when to switch fired at step 65 or 70, in every single run. So we had always been switching before the model learned, while explaining the saving with a story about how little work was left after it had learned. The saving was real and our account of it was on the wrong side of the event it named. The two results were never in conflict; they are two points on one curve, along an axis neither of them was varying.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 60 training runs, no GPU, no cost.

Program v2 Bucket Q, item Q8. Decisive computation: analysis/cut_reversal.py. Output: analysis/cut_reversal.json. Reproduce with python analysis/cut_reversal.py; --reuse re-derives every endpoint without retraining.

The question

Q1 found J7's batch cut reversing sign, a 22% saving in samples-to-target at width 48 on lag 4 became a 25.8% cost at width 192 on lag 11, and could not say why, because P12's matched ladder moves width and difficulty together by construction. A third candidate moves with them: scored tokens per step, which falls from 192 to 80 per cheap batch as the lag goes from 4 to 11.

**The shortcut helps before the model learns and hurts afterwards.** One of our results found that switching to a cheaper training setting partway through saved 22% of the data, and a later repeat on a bigger model found it costing 26% instead. We ran a grid varying model size and task difficulty separately to find out which was responsible. Neither was. Each bar is one combination, arranged by how far the switch was from the moment the model suddenly improved. Every switch made before the model's sudden improvement saved about a fifth of the data. Every switch made afterwards cost between nine and forty-two per cent. One combination shows it on its own: the same model on the same task, switched at two different times, saves in one case and costs in the other. Model size shifts how good the result is but never flips it, and the amount of data per step - the obvious explanation - decides nothing. The original result had always been switching early without realising it, and had explained itself with a story about switching late.

So: a 2x2 of {width 48, 192} x {lag 4, lag 11}, control against a switched arm, two switch times per cell, samples-to-target at a matched sample budget. No probe, Q1 showed a single fixed step matches the probe's timing, so it would add cost and variance and answer nothing.

Kill test, fixed before execution: the cut still saves at width 192, lag 4.

The anchors hold

	Here	Q1
`w48-lag4` control	`10,944` `[9,772, 12,116]`	`11,080` `[10,380, 11,780]`
`w192-lag11` control	`74,176` `[62,290, 86,062]`	`74,176` `[62,290, 86,062]`
`w192-lag11` switched	`90,288` `[81,574, 99,002]`	`92,080` `[85,936, 98,224]`

The middle row is identical rather than merely overlapping: same seeds, same code path.

The kill test does not fire, and it does not license the conclusion it was written for

The cut costs 42.4% at width 192 on lag 4, so by the letter of the preregistration the reversal is not "difficulty or scored tokens". But the grid does not say it is width either, and the reason is a confound of my own making: the switch is fixed at a fraction of the budget while the transition moves with width, so this design ties width to which side of the transition the switch lands on.

Two of the eight arms carry no information at all: their switch fell after the run had already reached the target, making them identical to the control rather than a null. They are excluded. The six that remain:

Cell	Switch vs its own transition	Scored tokens	Change
`w48-lag11`	`-156`	`80`	`+20.6%`
`w48-lag4`	`-20`	`192`	`+20.9%`
`w192-lag4`	`+29`	`192`	`-42.4%`
`w192-lag11`	`+59`	`80`	`-21.7%`
`w48-lag11`	`+369`	`80`	`-8.8%`
`w192-lag11`	`+584`	`80`	`-14.2%`

Every arm that switched before its own transition saved. Every arm that switched after it cost. Six for six.

Scored tokens per step decide nothing. 192 gives +20.9% and -42.4%; 80 gives +20.6% and three negatives. The hypothesis a practitioner would reach for first is dead.
Width does not decide the sign. Width 48 produces +20.6%, +20.9% and -8.8%.
Width does order the effect, and that is real: every width-48 arm sits above every width-192 arm (-8.8% against -14.2% at the join). Wider is worse at any given side. It is just not what reverses the manoeuvre.

One cell settles it on its own. w48-lag11, transition at 381: cut at 225 gives +20.6%, cut at 750 gives -8.8%. Same width, same task, same scored tokens, same budget, same seeds. Only the side of the transition differs, and the sign flips.

What this does to J7

J7's control transition, measured here on J7's own configuration and seeds, is step 88. J7's probe crossed at 65 or 70, in all eight seeds.

J7 always switched before the transition. Its 22% saving is the "before" effect, and its random-timed control drew from the same crossing distribution, so that arm switched before the transition too, which is exactly why the two matched.

And J7's stated mechanism was on the wrong side of its own event. Its reasoning was that at the probe's crossing "the answer is already linearly present in the hidden state and the head has not yet learned to read it", so the remaining work is cheap. That is a post-transition story about a switch that was consistently pre-transition.

Q1's crossings, 140 to 200 with a mean of 165 against a transition at 166, sit on the transition, which is why Q1 found no saving there. The two results are one curve, not a contradiction, and the axis is the one neither of them was varying.

Verdict

The kill test's literal outcome is reported as a failure, and separately from the conclusion, because a test whose grid ties two explanations together has not distinguished them. This is the same discipline P12's third case used.
Scored tokens per step are ruled out. That is a clean negative on the most obvious hypothesis.
Side of the transition explains all six arms; width explains none of the sign and all of the ordering.
A follow-up is needed and is cheap: switch at a fixed offset from each cell's own transition, which unties the two axes. Logged as Q11.

Limits

Two "before" arms against four "after". The split is 6 for 6 but the positive side rests on two cells, and the strong evidence is the within-cell pair in w48-lag11 rather than the tally. Q11 is the confirmation.
The transition is one of Q2's fourteen definitions: block_the_wrong_algorithm's, imported rather than re-implemented so no fifteenth is created. Q7's rule applies: the offsets here are 4 to 117 evaluation intervals from the transition, so none is inside the undecided band.
Five seeds per arm. One w192-lag11 run at the earlier switch finished at 0.4517 after reaching the target, so the cheap batch can destabilise a converged model on the harder task. That affects final accuracy, not samples-to-target, which is the endpoint here.
The two excluded arms are excluded on a structural fact, not a threshold: their switch step is later than the step at which the control reached the target, so those runs are the control.
One manoeuvre, one cut ratio, one crossing level, one task family. As in J7 and Q1, the 4x cut was fixed in advance and never swept, so "cutting the batch before the transition helps" is a statement about this cut.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
confound: A second explanation you did not control for. If bigger models both learn faster and score higher, then 'fast learners score higher' may be entirely about size and not about speed.
evaluation interval: The gap between successive checks of a model during training. It is the smallest difference in timing a measurement can resolve, so two results closer together than one interval cannot be told apart.
hidden state: The model's working memory: the internal numbers it carries from one step of a sequence to the next.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
preregistration: Writing down the design, the measurement and the decision rule before running the experiment, and not changing them afterwards.
probe: A small separate model trained to read information out of a bigger model's internals, used as a measuring instrument rather than as a product.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
variance: How spread out a set of numbers is. Rising variance in a signal means it is becoming more erratic.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.