How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

Checking Whether Our Result Was Only True of One Model

By Joshua Bolick, Awareness Software Group ~7 min read August 2026

Everything interesting we had found in our public research repository came from the same place: one model design, one task. That is a normal way for research to start and a bad place for it to stop, because the most common way a finding turns out to be wrong is that it was never a finding about learning at all. It was a finding about the specific thing being measured.

So we built four more model designs and ran every one of them on three tasks.

What we were looking for

When these models learn a task, they do it suddenly, and at that moment their internal state visibly reorganises. We have spent months characterising that reorganisation. The question here is simply whether it happens in other models, or only in ours.

Grid of model designs against tasks, shaded by the size of the internal reorganisation — Each cell is how much a model's internal state reorganises when it learns. Darker is more. Blank cells are combinations where the model never learned the task, so there was nothing to measure.

The good news

It generalises. A different gated design, the LSTM, shows the effect at full strength on every task it can learn. A deeper version of our original design shows it more strongly still. Nine out of nine models in that family show it, across all three tasks.

That matters because it means five months of downstream findings are not about one model. They are about a family of them.

The interesting news

Two designs show essentially none of the effect, and one of them is the design that most modern AI is built on: the transformer.

That result needed care to establish, because our first attempt at it was worth nothing and we knew it.

Why the first version of this test was worthless

Our original task asks a model to repeat a token it saw a few steps earlier. Looking back a fixed distance is precisely what attention does natively. A transformer does not have to learn a mechanism for that task; it arrives with one.

And indeed it solves the task almost instantly, and shows no reorganisation. But that proves nothing. No reorganisation was needed, so its absence is not evidence that the model cannot reorganise.

The fix is to give the transformer a task it cannot shortcut. We used one where the model is told at the start how far back to look, and has to vary that distance every time. One lookup is not the answer.

On that task, the transformer:

does learn it, to roughly the same accuracy as our original model;
takes longer to learn it than anything else in the entire grid, by a wide margin;
and still shows essentially none of the reorganisation.

That is a real negative rather than an empty one. Attention is genuinely doing something different, not merely being measured badly.

In plain English

Imagine testing whether athletes warm up before running. You test a sprinter, who does. Then you test someone on a bicycle, who does not, and you conclude that cycling requires no warm-up. But your test was a hundred metre sprint, and the cyclist just rode past the finish line without exerting themselves at all.

The fix is to make the cyclist actually work. Give them a long climb, watch them labour up it for far longer than the sprinter took, and observe that they still never warm up the way the sprinter does. Now you have learned something.

The explanation we had to throw away

The obvious story is that the reorganisation reflects effort: a model that has to work harder reorganises more. The grid refuses it outright.

The transformer taking the longest to learn shows the smallest reorganisation of anything measured. Our deeper model learns the same task in half the time with about ten times the effect.

How long the learning takes does not predict the effect. How the model is built does. The dividing line is not depth, and not whether the model processes things in sequence. It is whether the model has a particular gating mechanism in its memory. That was not what we expected to find, and it is now what we are studying.

One result we recorded as unknown

The fifth design, a simpler kind of memory, shows no effect on the easy task and cannot learn the harder ones at all. We could have counted it as a second negative. We did not, because on the easy task it has exactly the problem the transformer had: it solves it so fast there is nothing to observe.

Recording it as unknown rather than negative costs us a cleaner story. It is the same mistake we had already made once, and having made it once is a poor reason to make it again in the other direction.

What to take from this

If someone shows you a result about how an AI system behaves, the question that does the most work is: on how many different systems, and how different were they?

Findings from a single model, a single dataset, or a single configuration are where research starts. They are not where confidence should come from. Ours held up across a family and broke across families, and we only know which is which because we built the other four and ran them.

And when a test comes back negative, the follow-up question is whether the test could have come back positive. Ours could not, the first time. That version of the experiment was free to run and worth nothing, which is a combination worth watching for.

Read the full record, including the whole grid