How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

The Answer Was There Before the Model Could Say It

By Joshua Bolick, Awareness Software Group ~7 min read August 2026

There is a difference between a model knowing something and a model being able to tell you. In our public research repository we measured the gap, and it turned out to be about 23 training steps wide.

The more useful part of this story is that we got the answer wrong the first time, and the reason we got it wrong is a trap that applies to almost any measurement of an AI system.

Asking a model what it knows

A model's final step is a simple one: it takes its internal state and applies a straightforward mapping to produce an answer. That mapping is learned along with everything else, and it can lag behind.

So we asked a different question. At every step of training, we took the model's internal state and fitted the best possible version of that same final mapping, from scratch, on data the model was not being scored on. Not a cleverer method. Exactly the same kind of mapping the model itself uses, just fitted as well as it can be fitted.

Then we compared the two.

Two accuracy curves through training, with the readout curve consistently ahead of the model's own — One training run. The upper line is what can be recovered from the model's internal state by an optimally fitted readout. The lower line is what the model actually answers. The dashed line applies the same procedure to scrambled labels, and stays at chance as it should.

The upper line gets there first, consistently. It reaches 50% accuracy at step 59.3; the model reaches the same level at step 82.7. That is a lead of 23.5 steps, and it held in all ten runs we did.

It is not an artefact of picking 50% as the comparison point. We checked every level from 20% to 90%, and the lead was positive in every single run at every single level, ranging from 13 to 39 steps.

In plain English

Think of someone learning a language. There is a period where they understand a sentence perfectly well but cannot yet produce a reply. The knowledge is in there before the ability to use it shows up.

We measured that gap in a model, and it is real: about 23 steps of training during which the answer is sitting in the model's internal state, recoverable, while the model itself is still getting the question wrong.

Why this matters to us

Most of what you can measure about a model during training tells you what has already happened. This is one of the few things we have found that tells you what is about to happen. That has practical uses: knowing when a model is about to learn something means you can decide whether to keep spending money on it.

The part we nearly got wrong

The first time we ran this experiment, it produced the opposite result. A lead of 1.5 steps, with a margin of error that included zero, in one run out of ten. We recorded it as a dead end.

Nothing about the models changed between that version and this one. What changed was the readout we were fitting. It was too weak, in two separate ways, and both pushed the answer toward "no effect":

Wrong fitting method. We used a fast closed-form technique that is known to underperform at high accuracy. It finished training scoring six points below the model itself, which is a contradiction: a readout fitted optimally cannot lose to one learned by trial and error. That was the tell, and we nearly did not look at it.
Not enough data. The readout was being fitted on far too few examples for the number of parameters it had.

The consequence was specific and nasty. The under-powered readout got overtaken by the model before the point where we were timing the comparison. So its arrival time was late for reasons that had nothing to do with the model, and the lead we were trying to measure collapsed to nothing.

A weak instrument did not give us a noisy answer. It gave us a confident, clean, wrong one.

In plain English

Imagine testing whether a new smoke detector responds faster than an old one, using a stopwatch that runs slow. You would not get a vague result. You would get a precise, repeatable, wrong one, and nothing in the data would look unusual.

The only way to catch it is to check the stopwatch against something whose timing you already know.

The rule we adopted

We now have a standing rule in the repository, and it is the main thing we would pass on:

A negative result from a measuring tool is a statement about the tool, until the tool has been checked against something it is known to be able to detect.

For this experiment the check is concrete. At the end of training, an optimally fitted readout should match the model's own, because it is the same kind of mapping with more freedom. It does now, in 10 runs out of 10, at 0.9923 against the model's 0.9917. That check is reported next to the result, every time. Without it, the number is unsupported.

What to take from this if you are evaluating AI

"We tested it and found no effect" is a claim about two things: the system and the test. Most reporting only describes the first.

The question worth asking is what the test was demonstrated to be capable of detecting. Not what it failed to find, but what it was shown to find when the answer was already known. If a safety check, a bias audit, or a regression test has never been run against a case where the problem is definitely present, a clean result from it means considerably less than it appears to.

We are not above this. We published one negative result in this repository that was entirely an artefact of our own tool, and we caught it because a number that should have been impossible was sitting in our own output.

Read the full record, including both versions of the analysis

Asking a model what it knows

In plain English

Why this matters to us

The part we nearly got wrong

In plain English

The rule we adopted

What to take from this if you are evaluating AI

Want this measured on your data?