How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

Does Randomness Help AI Learn?

By Joshua Bolick, Awareness Software Group ~6 min read August 2026

Training an AI model is a partly random process. Every step it looks at a different random handful of examples, so the direction it moves in is never quite the same twice. That randomness is usually treated as a necessary nuisance.

There is a more interesting theory about it. Models we study do not improve smoothly: they sit at a low score for a long stretch and then jump, quite suddenly, to a high one. One explanation for that shape is stochastic escape. The model is stuck in a rut, and the randomness is what eventually knocks it out.

If that were true it would be immediately useful, because it would mean you could add randomness on purpose to make learning arrive sooner. So we tested it.

The first experiment said yes, clearly

The standard way to change how much randomness there is in training is to change how many examples the model looks at before each update. A small handful is noisy; a large one averages out and is much steadier.

So we ran the same total amount of data through models using handfuls of 16, 64 and 256 examples, and a fourth version that looked at every example it had before every single update, which is as close to no randomness as you can get.

The result was large, clean, and pointed exactly the way the theory predicted. The noisiest setting learned after 2,560 examples. The least noisy one took 34,202, more than thirteen times as many. More randomness, faster learning, right across the range.

That is the sort of result it is very tempting to stop at.

It was measuring the wrong thing

Changing the size of the handful does not only change the randomness. It also changes how many times the model gets to update itself. At a fixed amount of data, a model using handfuls of 16 takes sixteen times as many updates as one using handfuls of 256.

So the experiment had two explanations available and no way to tell them apart. More noise, or more practice?

Counting the same runs in updates rather than examples answers it. The ordering reverses. Measured that way the noisiest setting is the slowest, taking 160 updates against the steadiest setting's 33.

The apparent 13x noise effect was mostly the model simply getting more attempts.

The clean version

To ask the question properly you have to change the randomness and nothing else. So we did: same handful size, same number of updates, same everything, but with extra random noise added directly on top of each update, at four different strengths.

Bar chart showing learning arriving later as injected randomness increases — Each bar is how much data the model needed before it learned, at a different amount of added randomness. The leftmost bar is the untouched control. The whiskers show the spread across five independent runs.

Every increase in randomness made learning take longer. The strongest setting needed 70% more data than the control, and the effect was consistent across all four levels: a little noise did almost nothing, and a lot of noise clearly hurt.

The moment of learning also got blurrier. The jump smeared out over roughly twice the span, so noise did not just delay the event, it degraded it.

In plain English

Imagine trying to find a door in the dark. One theory says that stumbling around randomly is what eventually gets you through it, so you should stumble harder. We tried stumbling harder. It took longer to find the door, and we went through it less cleanly.

The randomness in AI training is not what solves the problem. Something else is doing that, and the randomness is getting in its way. Not enough to stop it, but enough to measure.

Why the direction is the whole result

Our own test for this experiment, written down before we ran it, was whether the moment of learning responds to noise at all. It does: well beyond the run-to-run variation we would expect by chance.

Read as a headline, "learning responds to randomness" sounds like support for the theory. It is the opposite. The theory needed noise to end the stuck period. What noise actually does is make it longer.

We had written the experiment's summary line to report the size of the effect. We changed it to report the direction as well, because the version without it was genuinely misleading about our own result.

What to take from this

A clean, large, consistent result can still be measuring the wrong variable. Nothing about the first experiment looked shaky. It was monotone across four settings and the effect was enormous. It was also mostly an artifact of a second thing changing at the same time.
"Responds to X" is not a finding without a direction. Half of the possible answers to "does it respond" support a theory and half refute it, and a summary that omits which is not a summary.
Adding noise is not a training trick. At least for this kind of model, deliberately increasing randomness costs data and sharpness and buys nothing. We had been holding that idea open as a possible line of work and this closed it.

The broader habit is worth stating plainly, because it is cheap and it keeps paying: when an experiment changes one knob, check what else that knob moves. Here the answer was hiding in a change of units.

Read the full record, with every number and the limits