How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

Three Buckets That Lied to Us

By Joshua Bolick, Awareness Software Group ~6 min read August 2026

Here is a result we did not publish, and the reason is worth more than the result would have been.

What we were measuring

When a model trains, each step nudges it in some direction. Those nudges partly cancel each other out, so the model ends up travelling less far than the total distance it moved. The cancelled part is wasted computing, and we wanted to know where in training it happens.

Specifically, whether the waste clusters around the moment the model suddenly learns the task. If it did, there would be an obvious place to look for savings.

The plan, written before we ran anything, was to group the measurements into three periods: before the moment of learning, around it, and after.

The three buckets

They came out like this, where a higher number means less waste:

Before: 0.949
Around: 0.884
After: 0.827

Cleanly ordered, with error bars nowhere near overlapping. By every normal standard that is a solid result: waste increases through the phases of training, and the moment of learning sits in the middle of the progression.

We could have written that up. It would have looked good.

What was wrong with it

The three buckets sit at three different points in time. That is what makes them buckets. So anything that changes steadily over training will separate them, whether or not it has the slightest connection to the moment of learning.

The comparison cannot tell the two apart. It produces a clean, ordered, statistically comfortable result in both cases.

So we threw the grouping away, measured at every single step instead, and lined each training run up on its own moment of learning before averaging.

A line falling steadily with no feature at the marked moment of learning — The same underlying data, measured continuously. Zero on the horizontal axis is the moment the model learned. A real phase effect would show a bump or a dip there.

The line falls steadily from beginning to end and does nothing whatsoever at the marked moment. We checked that directly rather than by eye: fitting a trend to everything outside the middle region, the middle region sits on that trend to within a third of the measurement noise.

The waste is real, and it is large. About a fifth of the model's effort stops counting between early training and the end of the window. It simply has nothing to do with learning the task.

In plain English

Imagine measuring people's height in three groups: primary school, secondary school, and university. The groups will differ enormously and the error bars will not overlap. You have not discovered that school phases cause growth. You have discovered that the groups contain people of different ages.

Our three buckets had the same problem. They were labelled by their relationship to an event, and separated by the fact that they sat at different times.

Why this is worth watching for

Before-and-after comparisons are everywhere: before and after a product launch, a policy change, a deployment, a treatment. They are intuitive, easy to explain, and they carry this problem built in whenever the thing being measured also drifts on its own.

Two things make it hard to catch.

The result looks better, not worse, when it is wrong. Grouping averages away the noise, so a spurious difference arrives with tight error bars and a clean ordering. Nothing in the output signals a problem.
The grouping usually comes from the plan. Ours was written in advance, which is normally the thing that protects you. Deciding a comparison ahead of time does not make that comparison capable of answering the question.

The fix was not clever and cost nothing extra: measure continuously, line the runs up on the event, and look at the shape. A real effect at the event appears as a feature. Drift appears as a slope. The buckets cannot distinguish those; the curve cannot confuse them.

What we reported instead

Two things, kept separate. That the waste is not connected to the moment of learning, which closed off a line of work we had been considering. And that waste grows steadily throughout training regardless, which is a real finding we would have missed entirely if we had stopped at the buckets and gone hunting for a mechanism at a moment where nothing was happening.

The honest version was more useful than the tidy one.

The full record, with every number and the limits