How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

We Tried to Kill Our Own Finding

By Joshua Bolick, Awareness Software Group ~6 min read August 2026

Our public research repository has one positive result. Almost everything else in it is a negative: an idea we had, a test we designed to kill it, and the record of it dying. The one survivor is a specific reorganisation that happens inside a model at the moment it learns a task.

Every experiment that produced that result used the same training algorithm, AdamW. It is the standard choice, we never varied it, and that meant the result might not have been about learning at all. It might have been about AdamW.

So we set out to break it.

Making the comparison fair first

You cannot simply swap the algorithm and rerun. Plain gradient descent at the same settings learns much later, and a later transition is a different transition. We would have been comparing two things at once.

So we tuned each algorithm's learning rate until all three learned the task at the same point, using a separate tuning run whose random seed was deliberately excluded from the measurement runs. Plain gradient descent needed a learning rate 320 times larger than AdamW's. All three then solved the task in 10 runs out of 10.

Half survived

Bars comparing effect size and timing consistency across three training algorithms — Blue bars: how large the internal reorganisation is under each algorithm. Orange bars: how often it happens at the same moment the model learns.

The size of the effect is essentially identical across all three: +0.2239 for AdamW, +0.2421 for plain gradient descent, +0.2273 with momentum. Within 8% of each other, and all three clear the threshold we had committed to in advance.

That is good news, and it is the more important half. The phenomenon shows up at full strength under plain gradient descent, which has no adaptive scaling, no momentum, and no per-parameter bookkeeping at all. It is not an artefact of a fashionable optimiser.

Half did not

The published version of this result had a second claim: the reorganisation happens at the same moment the model learns. Twelve runs out of twelve, within 50 steps.

That claim belongs to AdamW. Under AdamW the reorganisation lands 0.5 steps from the moment of learning, in 10 runs out of 10. Under plain gradient descent it lands 243 steps later, in 1 run out of 10. With momentum, 170 steps later, in 0 out of 10.

Several planned experiments wanted to use that timing as a clock, to trigger something at the right moment during training. Those experiments are now scoped to one optimiser until shown otherwise.

In plain English

Suppose you find that a particular engine noise reliably happens at the exact moment a car reaches highway speed. Useful: you could use the noise as a signal.

Then you test a different transmission. The noise still happens, just as loudly, so it is a real property of the car and not of the gearbox. But now it happens well after the car reaches speed. The noise was real. Its usefulness as a signal was a property of the gearbox you happened to be using.

A trap we had to avoid on the way

One detail is worth showing, because it is the kind of thing that quietly ruins comparisons.

We wanted a second measurement to back up the first. The obvious one was to measure the change from the moment of learning to the peak. That would have been useless here, because the peak lands almost immediately under AdamW and about 240 steps later under gradient descent. It would have compared a five-step window against a two-hundred-step window and called the difference a result.

We used a fixed 100-step window instead, identical for every arm. Every algorithm shows the effect on that measurement too. The first version of the analysis is recorded in the study document rather than quietly deleted, because how a measurement was wrong is often more useful than the corrected number.

Why we publish the half that failed

The tempting version of this article reports that our result survived a hard test, which is true. The complete version reports that a specific claim we had published did not, which is also true and considerably more useful to anyone deciding whether to trust the rest.

The same standard is what we would want applied to a vendor. When someone tells you a result is robust, the question is what they varied to find out, and what broke when they did. A robustness check that everything passes was probably not a robustness check.

Read the full record, including the code that produces every number