How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

What Moves Is Not What Matters

By Joshua Bolick, Awareness Software Group ~6 min read August 2026

There is a tempting way to work out which part of a system matters: watch it running and see which part is busiest. It is cheap, it is non-invasive, and in this case it gives the wrong answer.

Two questions that sound like one

Our research programme studies small models that learn a task suddenly rather than gradually. We wanted to know which internal component is responsible for that moment.

We asked it twice.

By watching. Measure how much each component's numbers change, and check whether any of them changes more than the others right at the moment of learning. One component stood out clearly, changing about eight times more than the quietest one.

By intervening. Hold one component completely fixed, never let it learn at all, and see how much worse the model does. Repeat for each component.

Paired bars showing how much each part changes and how much is lost when it is frozen — Blue bars: how much each part's numbers change during the moment of learning. Orange bars: how much accuracy is lost if that part is never allowed to learn. Each series is scaled against its own largest value.

They point at different components

The busiest component costs nothing to freeze. Hold it completely fixed and the model is, if anything, a fraction more accurate than before.

The component that actually matters changes less. Freeze it and the model loses 23 percentage points of accuracy, takes three and a half times longer to get there, and shows less than half of the internal reorganisation we had been studying.

Watching told us the wrong thing, confidently. The only way to find out was to hold each part still and see what broke.

In plain English

Picture diagnosing a machine by looking for the noisiest, hottest, fastest-moving part. Often that is the flywheel: it does a lot of visible work and the machine runs fine without it engaged. Meanwhile a small valve that barely moves is the thing everything depends on.

Activity and importance are different properties. Measuring the first and reporting the second is one of the easiest mistakes to make and one of the hardest to notice, because the answer looks so reasonable.

The second surprise: nothing was essential

We froze six different components, one at a time. Every single one still learned the task. The model routes around whatever you take away from it.

That is worth reporting because it would have been invisible under a simpler analysis. If we had recorded only pass or fail, every arm passes and there is no result. The finding lives entirely in how much each freeze cost, which ranged from nothing at all to nearly a quarter of the model's accuracy.

One clean pair

Two of the freezes line up with an earlier experiment in a way we did not anticipate.

Freezing the model's output layer, the part that turns internal state into an answer, costs nothing. A model whose output layer is left at its random starting values and never trained still reaches 99.5% accuracy.

But freezing the model's memory, which we tested separately, is devastating: accuracy roughly halves and learning becomes about nineteen times more gradual.

So the memory is doing the work and the readout is along for the ride. If you were deciding where to spend effort improving a system like this, that ordering is the useful output, and neither experiment gives it to you alone.

Why this generalises past our models

The pattern here is not specific to tiny research models. Any time someone tells you which part of an AI system is important, it is worth asking how they found out.

By watching? Attention maps, activation magnitudes, gradient sizes, which layers change most during fine-tuning. All of these measure activity. Our experiment is a clean case where activity and importance point at different components.
By intervening? Removing, freezing, or corrupting the component and measuring what breaks. More expensive, considerably more informative, and the only one of the two that answers the question actually being asked.

Both are legitimate. They are just not interchangeable, and the cheap one is the one usually reported.

Read the full record, including both halves of the experiment