How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

It Never Becomes Free

By Joshua Bolick, Awareness Software Group ~13 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Part of a bigger question: How do we know our own results are real? – Repeatedly, the control rather than the measurement decided the result -- and several striking findings dissolved when the right comparison was finally run.

In plain English

What we asked. Late in training, part of a model can sometimes be switched off permanently to save compute. We had one result showing this works: past a certain point, freezing the model's memory cost nothing measurable in quality while saving real work. It was the only positive efficiency finding we had. It was measured on one model size with a fixed task, which we now know is a setup that can invent effects that are not there.

What we found. Repeated on a model four times wider with a correspondingly harder task, it does not survive. The cost of switching off falls the longer you wait, from eighteen accuracy points down to three and a half, and never reaches the amount we had decided in advance we were willing to give up. The check that proves our measurement can see the effect at all passes overwhelmingly. And the timing is still worth nothing: waiting for each run's own moment does exactly as well as waiting the same fixed time in every run, at all four waiting times we tested.

Why it matters. The explanation is the part worth keeping. We had described the rule as 'wait a while after the model suddenly improves'. Checking the original runs at the moment they froze, the model had already reached 99.6% of the accuracy it would ever reach. It was not past some critical window, it was simply finished. On the larger model, the furthest point we tested still leaves a tenth of the improvement to come, and freezing stops the model getting it. The honest rule is 'freeze once the model has stopped improving', which needs no reference to the sudden-improvement moment at all, and which you can check directly. Two things had been true at once in the original setting, and we had credited the wrong one.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 71 training runs, no GPU, no cost.

Program v2 Bucket Q, item Q5. Decisive computation: analysis/freeze_matched.py. Output: analysis/freeze_matched.json. Reproduce with python analysis/freeze_matched.py; --reuse re-derives every endpoint without retraining.

The question

K2 is the only positive efficiency result in this programme. From 320 steps past the transition the recurrence can be frozen permanently for 7.1% of training cost at a paired accuracy change of +0.0011 [-0.0000, +0.0022]. Its second finding was the deflating one: a single fixed step, with no knowledge of the transition, matched the oracle to 0.0001.

**Switching off part of the model always cost something, however long we waited.** Late in training, part of a model can sometimes be switched off permanently to save compute. On smaller models we found a point past which this was free. Here the same test runs on a model four times wider, with the task made harder to match it. The shaded band is the amount of accuracy we decided in advance we were willing to give up for the saving. The cost keeps falling the longer you wait, and never reaches the band. On the larger model there is no point at which switching off is free, within a run three times longer than the original. The two lines also sit on top of each other: waiting for each run's own moment is worth nothing over waiting the same fixed time in every run, which is what the original experiment found too. Looking at when the model stopped improving explains it better than looking at when it suddenly improved.

Like J7 it was measured at width 48 on the fixed task, the condition P11 and P12 showed was confounded for six other results, and the condition Q1 has just shown can reverse a manoeuvre's sign.

Kill test, fixed before execution: the freeze saving's accuracy interval falls outside the materiality floor at every offset, or the fixed-schedule arm still matches the oracle-timed one to within noise.

The anchor reproduces K2 exactly

K2's configuration re-run through K2's own train(): width 48, lag 4, K2's eight seeds:

	Here	K2 published
paired accuracy at `+320`	`+0.0011` `[-0.0000, +0.0022]`	`+0.0011` `[-0.0000, +0.0022]`

Enforced as a precondition; no verdict prints without it.

Result: the kill test fires on both clauses

Width 192, lag 11, five seeds, 3,000 steps, freeze permanent. Mean control transition 161.0.

Arm	Final accuracy	Paired vs control	Saves
control	`0.9758` `[0.9696, 0.9821]`	,	,
freeze at `+160`	`0.7949`	`-0.1809` `[-0.2010, -0.1608]`	`21.7%`
freeze at `+320`	`0.8649`	`-0.1109` `[-0.1313, -0.0905]`	`20.4%`
freeze at `+640`	`0.9030`	`-0.0729` `[-0.0885, -0.0572]`	`17.8%`
freeze at `+1280`	`0.9405`	`-0.0354` `[-0.0460, -0.0247]`	`12.6%`
freeze at `transition-40` (positive control)	`0.7063`	`-0.2695` `[-0.2893, -0.2498]`	,
freeze at a random step	`0.8799`	`-0.0959` `[-0.1900, -0.0018]`	,

The positive control passes at -0.27, so the endpoint plainly sees freezing.

No offset in the sweep is free. The cost falls monotonically, -0.18, -0.11, -0.07, -0.035, and never reaches the 1% floor. K2's saving does not survive the matched ladder.

And the timing still buys nothing, now four times over. A single fixed step for every seed against each seed's own oracle:

Offset	Oracle	Fixed step	Gap
`+160`	`-0.1809`	`-0.1809`	`+0.0000`
`+320`	`-0.1109`	`-0.1111`	`+0.0002`
`+640`	`-0.0729`	`-0.0733`	`+0.0005`
`+1280`	`-0.0354`	`-0.0371`	`+0.0018`

Every gap is inside the floor by more than an order of magnitude. K2's second finding replicates at a new width and a new difficulty, at every offset tested.

What K2's boundary actually was

The obvious reading is that the critical period is longer at width 192, so the freeze has to wait longer. The data says something simpler and more useful.

	K2, width 48	Q5, width 192
freeze point	step `410`	step `1,441`
control accuracy at that step	`0.9881`	`0.8785`
control accuracy at the end	`0.9920`	`0.9758`
share of its final accuracy already reached	`99.6%`	`90.0%`

K2 froze after its model had essentially finished learning. +320 past the transition, at width 48 on lag 4, is a point where the control has 0.4% of its improvement left. At width 192 on lag 11 the furthest offset in this sweep still leaves 10% of the improvement to come, and freezing the recurrence stops the model getting it.

So K2's boundary was "past convergence", not "past the critical period". The two coincided at width 48 on lag 4 and separate here. That is the same shape as L7's finding and K2's own: a quantity defined against the transition, which happened to be measured where something else was also true.

This is not "freezing never works". The cost is still falling at the end of the sweep, so the boundary exists somewhere past +1280: it is simply past 48% of a run three times longer than K2's, and finding it needs a longer budget, not a different manoeuvre.

Q9's platform-bound diagnosis, confirmed by a second point

Q9 classified K2's 7.1% as platform-bound and predicted that a wall clock on a tiny model under a duty-cycle throttle measures framework overhead rather than the arithmetic being skipped. This pilot times a frozen step at both widths and prints the model beside it:

Width	Frozen step, timed	Arithmetic model	Factor
48 (K2)	`13.1%` cheaper	`60.0%`	`4.60x`
192	`24.3%` cheaper	`64.9%`	`2.67x`

The gap narrows as the model grows, from 4.6x to 2.7x, exactly as a framework-overhead explanation requires. Q9's classification is confirmed by an independent measurement, and the correct reading of K2's 7.1% is "at least 7.1%, on that machine".

Verdict

The kill test fires on both clauses. No offset is free at matched difficulty and width 192, and the fixed-schedule arm matches the oracle at every offset tested.
The programme's only positive efficiency result does not survive the matched ladder. Taken with L7 (K2 scoped to one task) and Q1 (J7's negative survives and strengthens), no efficiency claim in this archive currently stands under matched difficulty at a larger width.
The reframing is the reusable part. "Freeze N steps after the transition" was never the operative rule. "Freeze once the model has stopped improving" is, and it is checkable without any reference to the transition, which is another way of saying the transition was not what made K2 work.

Limits

The sweep stops at +1280 in a 3,000-step run, so it cannot locate the boundary, only establish it is further out than 48% of the run. A longer budget would find it and is the obvious follow-up.
Five seeds in the matched block against K2's eight. The random-timed arm's interval ([-0.1900, -0.0018]) is wide because its draws span 315 to 1,435; that width is a property of the control, not a defect, and it is reported rather than trimmed.
The 1% materiality floor is K2's, restated here rather than imported because K2 holds it as a local. Accuracy plateaus near 0.98 at both widths, so one point is a comparable fraction of achievable headroom at both.
**The saving column uses the timed per-step figure.** With the modelled 64.9% every saving is about 2.7x larger, and none of them is free, so the choice does not affect the verdict.
Width and difficulty move together on P12's matched ladder, so nothing here separates them, the confound named in Q8, which applies to this record as much as to Q1's.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
confound: A second explanation you did not control for. If bigger models both learn faster and score higher, then 'fast learners score higher' may be entirely about size and not about speed.
critical period: A stretch of time during which something has to happen for development to proceed normally. Borrowed from biology, where it describes windows in which a young brain must receive certain input.
freezing: Stopping one part of a model from changing while the rest carries on learning. A way of asking what that part contributes, by taking it away.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
plateau: A stretch of training where the model is not visibly improving. Often, though not always, followed by a sudden jump.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.