How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

An Average Is Not a Promise

By Joshua Bolick, Awareness Software Group ~7 min read August 2026

A while ago we found a way to tell, during training, that an AI model was about to suddenly get much better at its task. We wrote it up. One of the numbers in that write-up was 49 out of 49: in every single run we measured, the warning arrived before the event.

That number was correct. It also did not mean what it looked like, and the follow-up experiment that showed why is more useful than the original result.

What we did next

The obvious follow-up was to look for better warning signals of the same kind. We built twelve candidates, tested each in two directions, and set up the experiment so the results could not flatter themselves:

Every alarm was tuned against scrambled data first. Take a run, shuffle it so any real build-up is destroyed but the overall character is kept, and set the alarm's sensitivity so it goes off on no more than one in five of those. Otherwise you can make any alarm look early by making it twitchier.
We split the runs in two. Twenty-four tests at a one-in-five error rate will produce convincing nonsense by chance. So one set of runs was used to pick the winners, and a completely separate set, never looked at during picking, had to confirm them.

That second precaution earned its place immediately. In an early trial run, one candidate looked strongly positive on the picking set and reversed direction entirely on the confirming set.

Two of the twenty-four survived both.

The winner looked perfect

The best survivor fires about 19 steps before the model learns. Better still, the relationship between when it fires and when learning happens is almost exactly one to one. In this kind of work that normally means you have a dependable fixed warning window: however long the run takes, you get roughly the same amount of notice.

Then we looked at the individual runs.

Scatter plot showing most alarms arriving before learning but several arriving after — Each dot is one run. Dots below the dashed line are runs where the alarm arrived in time. Dots above it are runs where it arrived too late to be any use.

The alarm arrived too late in five runs out of thirty-one. Across the set, the warning time ranged from 61 steps too late to 70 steps early.

The average was real. The promise was not. And nothing in the usual summary, neither the average nor the near-perfect one-to-one relationship, would have told you that.

In plain English

A bus route with an average wait of five minutes might mean a bus every five minutes, or three buses at once and then nothing for a quarter of an hour. Same average, completely different thing to plan your morning around.

Averages are for reasoning about many cases. Guarantees are for deciding what to do about the one in front of you. Most reporting gives you the first and lets you assume the second.

Then we corrected ourselves

This left an obvious problem. Our earlier result said the same underlying signal worked in 49 runs out of 49. The new one said it fails about one time in seven. Both were measured carefully. Both could not be right about the same thing.

The tempting explanation was that the two experiments set their alarms differently, using two different rules for deciding how sensitive to be. That would have been a tidy story about one method being better.

It was also testable for free, because the comparison needs no new training runs: just apply both rules to the same set of runs and see whether they disagree.

They did not. Run side by side on identical data, the two rules gave the same average warning time to one decimal place and reliability within three percentage points of each other. The method was never the difference.

What the difference actually was

The runs.

Our original experiment had to discard about 40% of its runs for a technical reason: a warning signal needs some history to calibrate against, so runs where the model learned very early could not be tested at all. Every discarded run was a fast one. What survived was the slower half, and in a slow run a warning has the most room to arrive comfortably early.

So 49 out of 49 was a property of which runs were left, not of the method that measured them. On a set of runs not filtered that way, the same signal with the same detector gets it right about six times in seven.

Both write-ups now carry the correction. The original's average warning time and its main finding still stand; the reliability figure does not generalise, and it no longer claims to.

Why this is worth your attention

The specific numbers are ours. The shape of the mistake is everywhere.

Anything you had to exclude is a selection effect. Our exclusions were principled, documented, and had nothing to do with the outcome we were measuring. They still changed the headline number, because what gets excluded is rarely a random sample of what you started with.
"Works every time" deserves the question "out of how many, and which ones?" A perfect record on a filtered set is not a perfect record.
Test the tidy explanation before believing it. Ours cost nothing to check and was wrong. If we had reasoned about it rather than measuring it, we would have written a confident paragraph about detector design that was pure invention.

None of this needed new data or new equipment. It needed one more experiment after the one that produced the pleasing number.

The full record, with every number and the limits

The earlier result, now carrying the correction