How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research question

Is seeing it coming actually worth anything?

By Joshua Bolick, Awareness Software Group ~3 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Suppose you can tell a model is about to improve. Can you use that to save time or money?

Where this stands

So far, no. Both attempts to act on a warning were matched by acting at a random moment, and the detection cost more than the manoeuvre saved. What the warning is worth is a separate question from what makes training cheaper, which has its own page.

This is the question that separates an interesting measurement from a useful one, and it is the one most often skipped. An instrument has to be priced in the same units as its benefit.

Still open · 2 published results bear on this question.

What this does not settle yet

Stated plainly, because the gaps are as much a part of the record as the answers:

Every manoeuvre tried so far is a training-regime switch or a freeze. Is there a manoeuvre that genuinely requires knowing the moment?
No saving has been tested at a scale where the instrument's cost would amortise.
This question is deliberately narrow now: it asks only what the *warning* buys. The savings themselves live under what actually makes training cheaper.

How these results fit together

This question is deliberately narrow, and it is separate from whether anything makes training cheaper, which has its own page. It asks one thing: given a warning that a model is about to improve, can you *do* something with it that you could not do otherwise? Both attempts here say no, and they say it in the same shape. We switched training strategy when the probe fired, and got a real saving -- which was then matched exactly by switching at a random moment. So the saving was real and the *timing* contributed nothing. We then replaced the expensive detector with much cheaper ones, on the theory that the economics rather than the idea were the problem. The cheaper detectors worked, and the manoeuvre still did not beat random timing. That is the more informative of the two results: it rules out the explanation we would have preferred. The open thread is honest and specific -- every manoeuvre tried so far is a training-regime switch or a freeze, and neither genuinely needs to know *when*. A manoeuvre that does would put this question back in play.

The results

Each of these is a self-contained record: what we asked, what would have proved us wrong, what we found, and what it does not show. They open with a plain-language summary before any of the technical detail.