How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 43 of 47

We Found a Saving, and It Needed No Instrument

By Joshua Bolick, Awareness Software Group ~14 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. This project exists to find ways of making AI training cheaper, and after forty-seven experiments it had never actually made anything cheaper. This is the one that did. The most expensive part of these models to train is the part that carries information from one moment to the next, and we already knew it stops mattering for WHEN the model learns shortly after it has learned. So we asked whether you can simply stop training it at that point and pocket the saving.

What we found. You can, but not as soon as we expected, and the gap is the interesting part. Switch it off forty steps after the model learns and you lose about seven and a half accuracy points. Wait longer and the loss shrinks: three points, then under one, and by three hundred and twenty steps it is gone entirely. At that point you save about seven percent of the whole training run for nothing. So the part of the model that has stopped influencing when learning happens is still quietly improving how well it works, for a long time afterwards. Those are two different questions and we had been treating them as one.

Why it matters. Then we checked the part we were proud of, and it was not doing anything. Our timings were measured from the moment each individual model learned, which requires knowing that moment. But the safe point to switch off turns out to be so far past it that a single fixed schedule, applied identically to every run and using no knowledge of anything, works just as well, matching our careful per-model timing to four decimal places. The saving is real and worth having. The clever part of it was not. That is now the third time in this project that a genuine saving has turned out not to need the detection we built it around, which is starting to look less like a coincidence and more like the finding.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 64 training runs, no GPU, no cost.

Program v2 Bucket K, item K2. Decisive computation: analysis/freeze_after_transition.py. Output: analysis/freeze_after_transition.json. Reproduce with python analysis/freeze_after_transition.py, or re-decide the floor and boundary from the saved series with --reuse.

The question

D5 found the recurrence's critical period closes hard: freezing it at the transition costs +19.1 [+18.2, +20.1] steps, and freezing it forty steps later costs +0.1 [-0.2, +0.4]. After its window the recurrence is causally irrelevant to timing, while remaining the most expensive part of the model to train. F3 rules out freezing it from the start, a frozen recurrence never reaches full accuracy, which is precisely why nobody had looked at the other end.

**How long you must wait before switching off the expensive part.** The most expensive part of these models to train is the part that carries information from one moment to the next. We already knew it stops mattering for WHEN the model learns shortly after it has learned. This asks something different: how soon can you stop training it altogether and pocket the saving? Bars below zero mean the model ended up worse than if you had never switched anything off. Closer to zero is better. Switch it off too soon and you pay for it: waiting only 40 steps costs about 7.5 accuracy points. The cost shrinks the longer you wait and reaches zero by 320 steps, where the saving is about 7% of the whole training run for no loss at all. So the part of the model that has stopped affecting when learning happens is still improving how well it works, for a long time afterwards. Those are two different questions and we had been treating them as one.

**Knowing when the model learned added nothing to the saving.** Having found that switching off the expensive part is free if you wait long enough, the obvious next question is whether you need to know when the model learned in order to time it. The second bar uses that knowledge, measured separately for every run. The third ignores it completely and switches off at the same fixed point in every run. The fourth switches off at a random moment, as a check that timing matters at all. The second and third bars are identical to four decimal places. Detecting the moment the model learns contributes nothing here, because the safe point to switch off is so far past that moment that a fixed schedule lands in the same place. The random bar is worse, so the timing is not irrelevant, it is just that a calendar does the job as well as an instrument. Every compute saving this project has found so far turns out to be available without detecting anything.

Kill test, fixed in advance: at a compute-matched budget, freezing the recurrence after the transition either costs final accuracy beyond a headroom-scaled floor, or saves nothing once the trigger is priced.

The first correction: D5 does not license this

D5 froze the recurrence for a bounded window and then unfroze it. This item freezes it permanently, to stop paying for its gradients at all. Those are different interventions, and D5's +0.1 licenses "a 40-step freeze after the transition costs no timing", not "the recurrence can be abandoned from the transition onward".

The smoke run made the distinction concrete before any full run: freezing permanently at transition + 40 cost 5 to 9 accuracy points, while freezing at step 200+ cost nothing. So the question was never whether the manoeuvre works. It is from when, and a single offset can only answer yes or no, when the useful answer is a boundary.

Result: a boundary, not a yes

Eight seeds, matched budget of 57,600 samples, freeze permanent from the stated step. Paired against each seed's own control.

Arm	Final accuracy	vs control	Compute saved
control	`0.9904` `[0.9885, 0.9924]`	,	,
freeze at `transition+40`	`0.9153` `[0.9012, 0.9295]`	`-0.0751`	`11.2%`
freeze at `transition+80`	`0.9591` `[0.9511, 0.9672]`	`-0.0313`	`10.6%`
freeze at `transition+160`	`0.9821` `[0.9792, 0.9850]`	`-0.0084`	`9.4%`
freeze at `transition+320`	`0.9915` `[0.9897, 0.9934]`	`+0.0011` `[-0.0000, +0.0022]`	`7.1%`
fixed step (no oracle)	`0.9916` `[0.9896, 0.9936]`	`+0.0011`	`7.1%`
freeze at a random step	`0.9608` `[0.9056, 1.0160]`	`-0.0296`	,
freeze at `transition-40` (positive control)	`0.5585` `[0.5491, 0.5679]`	`-0.4320`	,

The positive control passes overwhelmingly. Freezing inside the critical period costs 43 accuracy points, so the endpoint plainly sees freezing, and a null at the boundary means something.

The cost falls smoothly with the offset and reaches zero. From +320 steps past the transition the recurrence can be frozen permanently for 7.1% of total training cost with no accuracy penalty: in fact a hair above control, consistent with mild regularisation, though the interval touches zero and nothing is claimed from it.

Freezing at a random step is not the same thing (-0.0296), so "freeze eventually" is not the finding. The random arm's interval is wide because its steps ranged from 95 to 753; it exceeds 1.0 at the top, which is an artifact of a t-interval on a bounded quantity and is reported rather than trimmed.

The second correction: the oracle contributes nothing

The offsets above are measured from each seed's own known transition, which is an oracle and not a usable method. The first run's verdict therefore ended with "pricing a real detector is the next question".

That was wrong, and the arithmetic says so. The boundary sits 320 steps past a transition whose own spread across seeds is 15 steps, the offset is 21x the spread. So transition + 320 and a single fixed step land within 7 steps of each other in every run.

A fixed-step arm was added, choosing the freeze point as 46% of the training budget with no knowledge of the transition at all. It matches the oracle to 0.0001:

+0.0011 (oracle) against +0.0011 (fixed step), difference -0.0001.

"Freeze after the transition" and "freeze late" are the same instruction here. The saving is real and it is free, and it is not evidence that the transition is worth detecting.

Verdict

The kill test does not fire, from a boundary. Permanently freezing the recurrence from +320 steps past the transition saves 7.1% of total training cost at no accuracy cost. This is the first positive efficiency result in the programme.
Earlier is not free. -0.0751, -0.0313, -0.0084 at +40, +80, +160. The recurrence goes on contributing accuracy long after its timing contribution has gone, which sharpens D5 rather than contradicting it: a critical period for when is not a critical period for how well.
And it needs no instrument. A fixed fraction of the budget does as well as an oracle. Taken with J7 and K1, every efficiency saving this programme has found is available without knowing when the transition happens. Three items designed to spend transition timing have now produced one real saving between them, and the timing contributed nothing to it.
The honest size. 7.1% of a small model's training on one synthetic task. It is a genuine saving and a modest one.

Limits

Eight seeds, one task, one width, one learning rate, four offsets. The boundary is located between +160 and +320; it was not bisected, so +320 is an upper bound on where freezing becomes free rather than the boundary itself.
The fixed-step arm's 46% was chosen after seeing where the boundary fell, so it demonstrates that a fixed step suffices, not that this fraction transfers. On a task whose transition varies much more than 15 steps the two arms would separate, and that is the condition under which the oracle would start to earn its keep.
The saving is measured as a wall-clock ratio in one process under a CPU duty-cycle throttle: freezing removes the recurrence's gradients but not its forward pass, measured at 16.4% of a step. The ratio is portable; the absolute milliseconds are not.
Accuracy sits near 0.99, so the 1% floor is a large fraction of the achievable headroom. A task with more room might resolve the +160 arm differently.
This is a compute saving, not a data saving. Every arm sees the same 57,600 samples; what changes is the cost of processing them.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
compute-matched: Comparing two methods by giving them the same amount of computing power rather than the same number of steps. A method that takes twice as long per step should not get twice the compute for free.
critical period: A stretch of time during which something has to happen for development to proceed normally. Borrowed from biology, where it describes windows in which a young brain must receive certain input.
freezing: Stopping one part of a model from changing while the rest carries on learning. A way of asking what that part contributes, by taking it away.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
regularisation: Anything added to training to discourage the model from a particular behaviour, usually to help it generalise.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.