How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

A Strong Correlation That Meant Nothing

By Joshua Bolick, Awareness Software Group ~6 min read August 2026

In our public research repository we had a question we thought was free to answer: does when a model learns predict how well it ends up performing? We had 183 archived experiments already sitting there. No new compute, no new runs, just a query.

The answer came back immediately and looked decisive. Across the forty-one eligible runs, the correlation between when a model learned and how accurate it finished was -0.679. Strong, negative, and exactly the kind of number that ends up in a slide deck.

It means nothing. Here is how we know.

One variable was driving both

The forty-one runs were not all the same size. They came in three widths: 36, 48 and 72. And model width drives both things we were correlating. Wider models learn earlier, and wider models finish more accurate. Pool them together and you get a strong trend that is really just width wearing a disguise.

Scatter of when each model learned against its final accuracy, coloured by model size — Each dot is one training run. Across all the dots there is a clear downward trend. The colours show why it is misleading: model size moves both axes at once.

Split the data by width and the finding evaporates:

Width 36: correlation -0.316
Width 48: correlation +0.503
Width 72: correlation -0.444

The signs disagree. One of the three points the opposite way to the pooled number that started all this.

Two more problems underneath

Even the individual pieces cannot support a conclusion, for two reasons that are worth naming because both are common.

The measurement was too coarse. Every archived experiment recorded its progress every ten steps. Within a single width, that leaves only two or three distinct values for "when did it learn", and between 72% and 75% of runs share the same one. A correlation computed over three tied values is a comparison of two or three groups wearing a correlation's clothing.

The outcome had no room to vary. Final accuracy spanned 0.0048 at width 72 and 0.0188 at width 48. When every model finishes between 99.0% and 99.5%, there is almost nothing for timing to predict.

So we recorded the result as not answerable from this data. Not a positive finding, not a null result. The measurement does not have the resolution the question needs, and saying so is the honest output.

In plain English

Suppose you notice that people who eat breakfast earlier tend to be healthier, and the correlation is strong. Then you notice your sample mixed shift workers with retirees. Age drives both the breakfast time and the health. Split by age group and the pattern falls apart, with one group even showing the reverse.

The original number was real arithmetic on real data. It just did not mean what it appeared to mean.

What did survive

The same analysis produced something usable, from the same runs. If width drives the timing so strongly, then timing should be predictable from the settings alone. It is.

Average step at which models of each size learn the task, with seed-to-seed spread — Bars are the average step at which each model size learns the task. Whiskers show the variation between runs that differ only in their random starting point.

The gap between sizes is about 39 steps. The variation between runs that differ only by random chance is about 5 steps. So the configuration determines the timing roughly seven times more than luck does, and the moment of learning can be predicted in advance from the settings to within about ten steps.

That is a real result, and it came out of the same afternoon as the one we threw away.

The buying lesson

Pooled numbers across mixed configurations are the single most common way an AI benchmark misleads, and it is almost never deliberate. A vendor runs a spread of model sizes, quantisation levels and prompt strategies, aggregates, and reports the trend. The trend is real in the aggregate and can reverse in every subgroup.

Two questions cost nothing to ask and are hard to answer badly:

What varied between the runs you pooled? If more than one thing varied, the headline correlation is not attributable to any of them.
How much room did the metric have to move? A claim of no quality loss on a benchmark where everything scores 99% is not evidence of anything. We wrote a separate article about that problem, because we had it too.

Read the full record, including the code that produces every number

One variable was driving both

Two more problems underneath

In plain English

What did survive

The buying lesson

Want this measured on your data?