How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

The Control That Cost Us Our Result

By Joshua Bolick, Awareness Software Group ~7 min read August 2026

Training a model means nudging millions of numbers, over and over, for a long time. An obvious question is whether all those nudges are worth the same. If some parts of a model repay adjustment far better than others, or if there is a moment in training when adjustment suddenly starts paying off, then you could spend your money there and skip the rest. That is not a small prize: training is most of what a model costs.

We measured it, and found a factor of sixteen. Then we ran two controls, and most of the finding went away. This post is about the controls, because they are the transferable part.

Measuring what a nudge is worth

The thing you want to know is not how far a model's internal numbers moved. It is how much its behaviour moved as a result. Those are different: you can shift a lot of numbers and change almost nothing about what the model does, or shift very few and change a great deal.

We did not invent a way to measure this. Another group published one earlier this year, along with a clever trick for computing it cheaply. We took their method, and then did something that is only possible because our models are deliberately tiny: we also computed the answer exactly, the slow and obvious way, and compared.

They agreed. The small disagreement that remained shrank in exactly the way random noise shrinks when you take more samples. Four times more samples, half the disagreement. That is the signature of a noisy measurement rather than a broken one, and it is the difference between using somebody's tool and trusting it. Checking an imported method against ground truth is cheap and almost nobody does it.

The finding

Effort is very unevenly rewarded. The best-placed group of internals returned about sixteen times as much change in behaviour, per unit of adjustment, as the worst-placed one. On its own that is a real opportunity: if sixteen-to-one differences exist, an optimiser that knew about them could stop spending evenly.

And it looked like the moment of learning mattered too. Our models sit near chance for a long time and then improve sharply, and measured at that moment every part of the model appeared to be earning three to four times what it earned early on. A concentration of useful work, right at the interesting event. That is the kind of result that gets written up.

Control one: measure a model that has not been trained

The claim being made is that some parts of a model repay effort better than others. The unspoken half is that this is something about the trained model, because otherwise it is a fact about the blueprint and not about learning.

So we measured the same thing on a model at the instant before its first adjustment. Untouched. The gap was seventeen times.

Slightly larger than the trained model's sixteen. The unevenness is built into how the model is wired and how its numbers are first set. Training does not create it, does not increase it, and there is therefore no moment during training at which to go looking for it.

Worth noting how close this came to passing. Our first version of this control measured the model at the first point we happened to be recording, which was ten adjustments in, and therefore already trained. It gave a number that supported the same conclusion, by luck. A control that is almost right is not a control.

Control two: compare against a moment picked at random

The second claim was that useful work concentrates at the moment of learning. We had compared that moment against early training and found three to four times more. The trouble is that early training is not the right comparison. If the quantity simply climbs the whole way through, then any late moment beats any early one and the moment of learning is doing no work in the explanation.

So we took the same measurement at a moment chosen at random. The random moment scored higher than the moment of learning. Once the two are compared directly, the moment of learning comes out at 0.72 to 0.85 times the random one, with error bars comfortably spanning "no difference at all".

The quantity does climb through training. That climb is real. It is just not an event, and without the random comparison it would have been reported as one.

Why this keeps happening

This is the fourth time in this project that a control has turned an apparent event into an ordinary trend. The pattern is consistent enough to state as a rule:

Any claim that something happens at a moment needs a randomly chosen moment beside it. Comparing your moment of interest against the beginning tests whether anything changed over time, which is almost always yes, and is not the question you asked.
Any claim that training produced something needs an untrained model beside it. Architectures have properties before they learn anything.
The controls have to be exactly right, not roughly right. Both of ours were subtly wrong on the first attempt, and both wrong versions supported the conclusion we expected.

What survived

Not nothing, and the split is useful. The sixteen-fold unevenness is real; we only learned that it is a property of the design rather than of training. That means an approach which exploits a fixed, known difference between parts of a model is untouched by any of this, and it needs no timing, no detector and no monitoring, which makes it cheaper than the thing we were originally imagining.

What died is the version that needed to find the right moment. We have now closed that shape of idea several times over, from several directions.

There is also a number attached to the instrument. Running the measurement at the cadence its authors suggest costs about a tenth of the training budget. We report that alongside the result as a matter of course, because a measurement that costs more than the saving it finds is not a saving, and it is remarkably easy to publish one without noticing.

{_repo( "experiments/studies/function-space-work-v1.md", "The full record, with every number, both controls and the limits", )}