How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

When a Benchmark Is Too Easy to Be Useful

By Joshua Bolick, Awareness Software Group ~6 min read August 2026

For about a year, our public research repository ran experiments that each carried a quality check: did this intervention hurt the model's accuracy? Every single one passed. Not one intervention cost measurable quality.

That reads as a clean run of results. It is partly an artefact, and the reason is embarrassing enough to be worth writing down.

Nothing could have failed

The task we were testing on scored between 96% and 99.5%. A model with 0.4 percentage points of room above it cannot lose 2 points. So "no quality loss" was in part a statement about the task, not about the intervention.

It is not that the checks were wrong. They were correctly computed, correctly reported, and incapable of detecting the thing they existed to detect. A test that everyone aces does not rank anyone.

In plain English

Imagine screening pilots with an eye test where the smallest line on the chart is enormous. Everybody passes. The test is administered honestly and the results are recorded accurately, and it still tells you nothing about anyone's eyesight.

That was our quality check. To find out whether something degrades a model, the model has to be able to visibly get worse.

Building a harder test, and failing twice

So we designed replacement tasks with deliberate difficulty, and set the acceptance rule before running anything: a five-run control has to settle between 60% and 90% accuracy, and it has to actually settle rather than pass through that range on its way to solving the task.

Four candidates. Two passed.

Bars showing where models settle on each candidate task, against a shaded target range — Where models settle on each candidate task. The shaded band is the target: hard enough to leave room for damage to show, easy enough to be learnable. Green was adopted, orange rejected.

The two failures were more informative than the successes.

One task was all or nothing. Its five-run average was 0.6486, comfortably inside the target band, and it would have been adopted on the average alone. The second half of the rule caught it: individual runs landed anywhere from 0.491 to 0.974. Some solved the task outright and some did not, and the average was an artefact of mixing two different outcomes. Nudging the difficulty did not help. It went from 96.4% to 33.6% between two adjacent settings, with nothing in between. A task like that is a bad instrument, because a small change in conditions produces a large change in score for reasons unrelated to what you are measuring.

One task was trivial. It scored a perfect 1.0000 in every run at every setting we tried. The model architecture we use has a natural counter built into it, so asking it to count creates no difficulty at all. We recorded that as a boundary rather than tuning until something passed.

We also included a calibration task whose ceiling is known by arithmetic, at 0.8547. The measurement came in at 0.8491, an error of half a percentage point. That is what makes the other rows believable: the instrument was checked against a known answer before being trusted on unknown ones.

Quality claims in a unit that means something

With tasks that have room to fail, we changed how quality is reported. Instead of an absolute drop in accuracy, we now report the share of the available range that was given up.

Losing half a percentage point on a task that tops out at 99.6% is 0.5% of the available range. Losing the same half point on a task that tops out at 61.8% is 0.9%, nearly twice as costly, because there was less room to begin with. Same number, different meaning, and only the second version is comparable across tasks.

What to do with this when you are buying

"No measurable accuracy loss" is one of the most common claims in AI procurement, and it is only as strong as the headroom in the benchmark behind it. Three questions:

What does an unmodified baseline score on this benchmark? If the answer is above about 95%, a no-loss claim is close to unfalsifiable.
What would a deliberately damaged version score? If nobody has run that, nobody knows whether the check can detect damage.
Was the pass mark set before or after the results came in? Ours is written into the script, above the code that applies it, and it is the reason two of our four candidate tasks were rejected.

We are not immune to this. We ran a year of experiments on a saturated benchmark and every result was reported accurately. Accurate reporting of an insensitive measurement is still an insensitive measurement.

Read the full record, including the acceptance rule and the code that applies it

Nothing could have failed

In plain English

Building a harder test, and failing twice

Quality claims in a unit that means something

What to do with this when you are buying

Want this measured on your data?