How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 45 of 54

The Training Curve Cannot Tell You

By Joshua Bolick, Awareness Software Group ~12 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. When one of these models suddenly gets good at its task, something reorganises inside it. We had argued that this reorganisation is the model building the machinery it needs, rather than tidying up machinery it already had. That is a claim with a sharp consequence: a model handed the finished machinery should not need to reorganise at all. It is also the version of this question that matters commercially, because almost nobody trains a model from nothing. They start from an existing one.

What we found. So we trained models until they solved the task, then started fresh ones from pieces of them. A model given the finished output stage reorganised exactly as much as one built from scratch. A model given the finished internal machinery barely reorganised at all. The claim holds, and the second comparison is what makes it mean something: if both had gone quiet, the result would have been about starting from an existing model in general rather than about the machinery specifically.

Why it matters. The uncomfortable part was not what we set out to test. The model given the finished machinery still shows a sudden jump in its accuracy, and it learns about four times faster. So if you were watching its training curve you would see the same shape you see when a model builds something from nothing, and you would have no way to tell the two apart. That is worth knowing if you are paying for a model to be fine-tuned on your data: an impressive-looking training curve is consistent with the model having learned something new, and equally consistent with it fitting a thin layer onto capability it already arrived with. Only measuring inside the model separates them.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 30 training runs, no GPU, no cost.

Program v2 Bucket K, item K7. Decisive computation: analysis/warm_start.py. Output: analysis/warm_start.json. Reproduce with python analysis/warm_start.py, or re-derive the endpoints with --reuse.

The question

Every measurement in this programme is of learning from scratch. F3 says the transition creates features rather than selecting them, which makes a sharp prediction nobody had tested: hand a model an already-solved recurrence and a fresh head, and there should be no transition, because the mechanism the transition builds is already built.

**Give the model the finished mechanism and the reorganisation stops.** When one of these models learns, something reorganises inside it. We had argued that this reorganisation IS the model building the machinery it needs, rather than polishing machinery it already had. If that is right, a model handed the finished machinery should not need to reorganise. So we trained models to solve the task, then started fresh ones from parts of them. The middle bar is given the finished output stage; the right-hand bar is given the finished internal machinery. Give a model the finished output stage and it reorganises exactly as much as one built from scratch. Give it the finished internal machinery and the reorganisation almost entirely stops. So the reorganisation really is the machinery being built. The practical use is a warning: the model given the machinery still shows a sudden jump in its accuracy, and learns four times faster, so the visible training curve cannot tell you which of the two is happening. Only looking inside can.

Kill test, fixed in advance: warm-started runs show a transition of the same magnitude as from-scratch ones, on J4's expansion endpoint in a stated frame.

Result

Six seeds, each with its own donor trained to solution. Expansion measured at its best window anywhere in the trajectory (I6's generous endpoint), in both frames because D6 showed the moving-basis quantity mixes rotation with expansion.

Arm	Start	Final	Transitioned	Expansion (moving)	Expansion (frozen)
scratch	`0.035`	`0.9876`	`6/6`	`+0.2474` `[+0.2342, +0.2605]`	`+0.4896` `[+0.4736, +0.5055]`
warm-recurrence	`0.056`	`0.9933`	`6/6`	`+0.0410` `[+0.0199, +0.0621]`	`+0.1800` `[+0.1502, +0.2099]`
warm-head	`0.034`	`0.9904`	`6/6`	`+0.2812` `[+0.2680, +0.2944]`	`+0.5141` `[+0.5008, +0.5274]`
warm-all	`0.983`	`0.9932`	`0/6`	`+0.0422`	`+0.0583`

The kill test does not fire.

Zero is the wrong reference point, and `warm-all` says so

The warm-all arm starts solved, never transitions, and still registers +0.0422 and +0.0583. Some of what a best-window endpoint reports is drift that happens to any model being trained, not an event. So every arm is measured against that baseline rather than against zero:

Arm	Share of the from-scratch expansion retained (moving)	(frozen)
scratch	`100%`	`100%`
warm-recurrence	`-1%`	`+28%`
warm-head	`+116%`	`+106%`
warm-all	`0%`	`0%`

Handing over the recurrence removes essentially all of the expansion. Handing over the head removes none of it.

A materiality test alone would have missed this. It calls warm-recurrence's frozen +0.1800 "material" and stops, when the informative fact is that it is a fraction of scratch's +0.4896 and sits close to a baseline that involves no learning event at all.

The converse control is what makes it a claim about the recurrence

If both warm arms had been smooth, the result would have been about warm-starting in general. They are not: warm-head retains 116% and 106%, indistinguishable from scratch, and it transitions in 6/6 runs. The effect is specific to which part is handed over.

And the transplant demonstrably works: warm-all starts at 0.983 and never transitions, which is what proves the weights are really being copied. A warm arm starting at chance would have meant the transplant was a no-op and every other number here was noise.

An unplanned finding: mechanism transfer is worth 4x

warm-recurrence still shows an accuracy jump, but at step 20 against scratch's 85. Given the mechanism, fitting the readout takes a quarter of the time. The accuracy curve still has a step in it; what has gone is the internal reorganisation underneath.

That distinction is the practical content of this record. A jump in the loss curve does not by itself tell you a model is building anything.

What nearly went wrong

The analysis window was inherited from I6 and J4, which take the frozen basis at step 100 because their transitions are near step 150. warm-recurrence transitions at step 20. At FREEZE_AT = 100 the frozen frame would have been established eighty steps after the event it exists to measure across, and the analysis window skipped the first hundred steps entirely, so the one arm the experiment exists to examine would have been measured over the wrong stretch of its own run. Caught in a smoke test, and the constants are now chosen for the fastest arm rather than the slowest.

Verdict

The kill test does not fire. F3's prediction holds. The expansion accompanies building the mechanism, not fitting the readout.
In direction and magnitude, not in its strong form. F3 predicts no transition; what is observed is -1% of the expansion in the moving frame and +28% in the frozen one. Something survives.
"Does this model show a transition" is now a diagnostic, which is what the backlog hoped for: it distinguishes a model building a mechanism from one fitting a readout onto a mechanism it already has. That is the most transferable thing in this programme for anyone fine-tuning a real model, where warm-starting is the normal case rather than the exception.
And the accuracy curve alone will not tell you. The warm arm still jumps; only the internal measurement separates the two cases.

Limits

Six seeds, one task, one width, one learning rate, one donor per seed.
The donor is a solved model of the same architecture on the same task. Real warm-starting transfers from a different task, and nothing here speaks to partial or imperfect transfer.
warm-recurrence copies four tensors (both recurrent weight matrices and both biases). It does not isolate which of them carries the effect; J3 and K5 suggest the recurrent-to-recurrent matrix is the load-bearing one, but this does not test that.
The +28% in the frozen frame is unexplained. It may be the readout reorganising the representation it reads from, or residual drift the warm-all baseline under-counts.
Best-window endpoints are positive under noise by construction, which is why the baseline arm exists. It is a correction, not a proof that the remainder is signal.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
baseline: The thing you compare against. A result without one is not a result.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.