How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research question

What actually makes training cheaper?

By Joshua Bolick, Awareness Software Group ~4 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Training a model costs money and time. This is where we test the ideas that are supposed to reduce that cost, and report what they actually save once the cost of running them is counted.

Where this stands

One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

This is the question the whole project exists to answer, and the one where it is easiest to fool yourself. A saving is only real once you have subtracted what the method cost you, in the same units, and compared it against the simplest thing that could have worked instead.

Still open · 4 published results bear on this question.

What this does not settle yet

Stated plainly, because the gaps are as much a part of the record as the answers:

The one real saving came from a fixed schedule, not from measuring anything. We do not yet know of a saving that requires knowing what the model is doing.
Nothing here has been tested at a scale where an instrument's cost would amortise, or on anything other than a small model on a synthetic task.
Bucket O is mostly aimed at this question: progressive width, function-space accounting, and whether any of it survives a compute-matched control.

How these results fit together

This is the question the project exists to answer, and it is the one where it is easiest to fool yourself, so the standard here is deliberately harsh: a saving counts only once you have subtracted what the method cost you, in the same units, and compared it against the simplest thing that could have worked instead. By that standard one thing has worked. Freezing the expensive recurrent component late in training saved about 7% of the run with no loss of quality. The instructive part is that timing it from the model's own behaviour gave no advantage over picking a fixed step in advance -- the saving is real and the measurement that produced it was not needed. That has now happened three times in different guises, and it has become a standing rule here: find the best offline schedule before inventing anything adaptive. Two attempts failed in a way worth reporting. Choosing training data cleverly barely beat taking whatever came first, even with an unrealistically expensive selector. And a method that looked fastest turned out to be winning because it was quietly being given more than its competitors -- a reminder that most efficiency claims die on their controls, not their ideas.

The results

Each of these is a self-contained record: what we asked, what would have proved us wrong, what we found, and what it does not show. They open with a plain-language summary before any of the technical detail.

Back to all research questions

What actually makes training cheaper?

Where this stands

What this does not settle yet

How these results fit together

The results

We Found a Saving, and It Needed No Instrument

Predicting It Without Watching

Being Choosy About Data Costs More Than It Saves

The Method That Looked Fastest Was Cheating

Want this measured on your data?