How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

What a Model Has to Build, and What It Gets for Free

By Joshua Bolick, Awareness Software Group ~6 min read August 2026

A useful question about any AI system is which parts of it are doing real work. Not which parts are large, or expensive, or fashionable. Which parts, if you stopped training them, would actually break something.

In our public research repository there is a cheap way to ask that: freeze a part and see what happens.

The experiment

The models we study have three pieces. An input layer that turns tokens into numbers, a memory that carries information forward through a sequence, and an output layer that turns the memory's state into an answer.

We froze the memory at its random starting values and trained only the input and output layers. The model could learn to read its memory as well as it liked. It just could not change how that memory worked.

This is a known technique with a name, reservoir computing, and it sometimes works surprisingly well. We had a specific reason to expect it might: in a separate experiment we found that even an untrained memory carries real information about the right answer, about five times better than guessing.

Two accuracy curves against model size, one for normal training and one with the memory frozen — Final accuracy at five model sizes. The upper line trains everything. The lower line leaves the memory fixed at its random starting values and trains only the input and output layers.

It never catches up

Normal training has essentially solved the task by the third size, at 99.5% accuracy. The frozen version reaches 30.6% at that size. Making the model four times larger gets the frozen version to 57.4%, and normal training is at 99.97% there.

The gap never closes at any size we tested.

The part we would have got wrong

Stopping there would have produced a tidy and slightly misleading article, and the reason is a measurement rule we hold ourselves to.

A frozen model is cheaper to train per step, because there is less to compute. Comparing the two at the same number of steps therefore gives the frozen version less total compute, and quietly stacks the comparison. Our rule is that efficiency comparisons must be matched on compute, not on steps.

So we gave the frozen version four times the training budget. It changes the conclusion in an interesting way.

With four times the training, the frozen model does eventually clear the bar we use to say a model has learned the task. So "it cannot learn" would have been the wrong claim, and it is the claim the first version of this comparison supported.

What it cannot do is learn suddenly. Normal training goes from a quarter to three quarters of its final ability in 63 steps. The frozen version takes 1,175 steps to cover the same stretch, and still ends at 47% rather than 99.5%. That is about nineteen times more gradual.

In plain English

Think of the difference between rearranging your kitchen and learning to cook in someone else's. In your own kitchen you can move things, and once you do, everything gets fast at once. In a kitchen you cannot rearrange you will still improve, by memorising where everything already is, and you will improve steadily rather than suddenly, and you will never get as fast.

The sudden part of learning, in these models, requires being able to move the furniture.

What we take from it

Two things, and the second matters more than the first.

Random structure gives a real head start, and it is not enough. An untrained memory carries usable information, roughly ten times better than guessing once you attach a trained readout. That is genuinely something. It is also nowhere near a working system, and the distance between the two is most of the problem.

Reaching the same place is not the same as getting there the same way. Our original comparison used a simple pass or fail: did the model learn the task? With enough budget, the frozen version passes. Only when we measured how abruptly each one improved did the actual difference show up, and it is a factor of nineteen.

That second point generalises past this experiment. A benchmark score is a snapshot of where a system ended up. Two systems with the same score can have completely different shapes behind them, and the shape is often what you actually care about: how much data it took, how suddenly it became useful, whether it will keep improving or has flattened out. If you are comparing AI systems on a single final number, you are comparing the destinations and ignoring the routes.

Read the full record, including the compute-matched comparison

The experiment

It never catches up

The part we would have got wrong

In plain English

What we take from it

Want this measured on your data?