How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

The Price We Quoted for Our Own Measurement

By Joshua Bolick, Awareness Software Group ~8 min read September 2026

If you monitor a training run, the monitoring is not free. It runs on the same hardware, competing with the thing it is watching. So any claim that monitoring saved you money has to subtract what the monitoring cost, in the same units as the saving. We had made that a rule after learning it the hard way.

We had not made it a rule that the units have to be capable of changing.

The number

One of our results uses an early-warning signal: a small extra model, trained alongside the real one, that can tell when the real one is about to improve. We measured what running it costs and published the figure. 55% of the training budget. An enormous overhead, and the result was that the warning was not worth what it cost.

That figure then travelled. It appears in eight of our published results as the reference price for what a measurement costs, in write-ups that compare different model sizes and different tasks.

Here is the problem. The 55% was computed by counting rows of scored data: rows of monitoring against rows of training. Write out the ratio and the rows cancel. What is left is

when the warning fires x monitoring batch size / (how often we check x total data)

The model's size does not appear anywhere in it. No matter what we did to the model, that number could only respond to how long we trained and when the warning happened to fire. It was being quoted in comparisons across model size, and it was structurally incapable of answering one.

What it actually costs

Counting the arithmetic instead, counting the multiplications a training step performs against the ones a monitoring step performs, gives a very different picture, and it is a picture rather than a number.

Line chart of monitoring cost per model size against the single quoted figure — What our early-warning measurement costs at each model size, expressed as a multiple of the training it is watching. The dashed line is the single figure we had been quoting.

On our smallest models the monitoring costs about fifty times the training it is watching. On our largest, about seven times. Two facts matter here and the single published number could show neither:

It gets dramatically cheaper as models grow, by about sevenfold across the range we tested. That is real, and it is the direction that would eventually make this kind of monitoring practical.
It never becomes cheaper than the thing it is watching, at any size we ran. Not once.

The published 55% was, in arithmetic terms, closer to 190% at the size it was measured: the monitoring cost nearly twice the entire training run. Our conclusion had not been wrong; it had been wrong by less than we thought.

And the shortcut it was supposed to buy has reversed

The point of the early warning was to trigger a shortcut: partway through training, switch to a cheaper setting. On our smaller models that reached the target on 22% less data, though switching at a completely random moment worked just as well, so the warning itself was contributing nothing.

We have since learned that measuring something at a single model size, while holding the task fixed, quietly makes the model oversized for its job and can invent effects that are not there. So we repeated the whole thing on a model four times wider, with the task made correspondingly harder.

Bar chart showing the shortcut now costs more, and all three timing strategies land together — Training data needed to reach the target. The shortcut now needs about a quarter more data than not taking it, and switching on the warning, at random, or at a fixed step all land in the same place.

The shortcut has reversed. It now needs about a quarter more data than simply not taking it, and the gap is larger than our measurement uncertainty. And the timing is still worth nothing, now with twice as much warning as before: firing on the signal, firing at random, and firing at the same fixed step in every run all land in the same place.

So the two halves moved in opposite directions. The instrument got seven times more affordable; the thing it was buying went negative. An instrument becoming affordable is not the same as its output becoming worth buying.

The error that runs the other way

Recounting our costs turned up a second one, and it makes a result of ours better rather than worse.

We have one positive efficiency finding: past a certain point in training, part of the model can be switched off permanently at no cost to quality. We measured the saving with a stopwatch, timing a normal step against a switched-off one, and reported 7%.

On models this small, a stopwatch mostly measures our software framework. The actual arithmetic being skipped is about 60% of a training step, not 13%. Our own earlier work had already noticed the same thing from the other side: we had measured how training time grows with model size and got an exponent of 1.08 where the theory says 2, which is the signature of a measurement dominated by fixed overheads rather than by the calculation.

So our one positive result is understated by roughly a factor of five. Where arithmetic dominates, which is to say at any size anyone would actually care about, the saving should be far larger than we claimed.

An understatement is as much an error as an overstatement, and it is much less likely to be caught. Nobody re-examines a result that looks modest. A 7% saving attracts no scrutiny at all; a 60% one would have been checked by everybody, including us.

Three things worth taking away

Price a measurement in a unit that can respond to whatever you are varying. A ratio whose unit cancels the axis under test will answer a different question, confidently, and nothing about it looks wrong.
A stopwatch on a small system is a measurement of your framework. It is honest for that machine and it is not a property of the method. Put the arithmetic estimate beside it; the two disagreeing by five times is the finding.
Check where a number travelled before you correct it. Ours was in eight documents. Fixing the page it came from does not reach the seven that quote it, and a number that has become a reference price is exactly the kind that spreads.

None of this changes what our models are: small, synthetic, and limited in ways stated on every page. What it changes is that two of our own numbers were wrong in opposite directions for the same underlying reason, and neither would have been caught by checking the experiments, only by checking the arithmetic behind the accounting.

The full record, with every figure, its unit and its classification

The number

What it actually costs

And the shortcut it was supposed to buy has reversed

The error that runs the other way

Three things worth taking away

Want this measured on your data?