How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Right Order, Right Answers, No Room

By Joshua Bolick, Awareness Software Group ~12 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. Two earlier experiments tried to let a small thinking model decide how long to think about each question. One thought in the wrong places; the other thought in the right places but gave confidently wrong early answers, because those answers had never been trained. This time we trained the model to give a proper answer at every step of thinking, then let it stop as soon as it was confident.

What we found. That fixed both problems at once. The model spent about two steps on the easiest questions and over three on the hardest, and the answers it stopped on were right 99% of the time. But it only matched a fixed budget of three thinking steps for every question; it did not beat it. On this puzzle, three steps is already nearly perfect for everything, so there was nothing left for smarter stopping to win.

Why it matters. The lesson: a model that decides how long to think needs its answers trained at every point where it might stop, and even then it can only pay off where a single fixed amount of thinking leaves something on the table. Check that first.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, S3's six seeds. The design and kill test were committed (9f6377a) before any run.

Program v2 Bucket S, item S13. Decisive computation: analysis/supervised_step_halting.py. Output: analysis/supervised_step_halting.json. Reproduce with python analysis/supervised_step_halting.py (a few minutes); --reuse re-derives every endpoint.

The question

Three records now, one thread:

**Three ways to let a model decide how long to think, against a fixed budget.** A small model that thinks for a variable number of steps before answering, with three different rules for when to stop, compared with simply fixing the number of steps. Each dot is one setting of a rule. A penalty keeps accuracy but thinks in the wrong places; stopping on confidence thinks in the right places but its early answers are wrong. Training every step to give an answer fixes both, and those dots sit on the grey line. They match a fixed budget rather than beat it, because on this task three steps is already nearly perfect: there was nothing left to win.

S3: a thinking-step penalty removes 38% of a small model's thinking for at most a point, and a fixed budget of three steps beats every penalised model, because the learned halting unit is inverted -- most thinking on the inputs it never gets wrong.
S9: stopping on answer confidence turns the order the right way round and collapses accuracy, because the model's early answers were never trained and come out confidently wrong.
S9's conclusion: a halting signal needs to be both ordered by difficulty and calibrated at every step, and neither S3's nor S9's was both.

S13 builds one that is both, as cheaply as possible: the same cell, trained so every thinking step's answer is supervised at equal weight (deep supervision), 4000 steps like S3's arms, then stopped on confidence at S9's eight thresholds.

Kill test, fixed before execution: at matched mean thinking steps, it does not beat S3's fixed-budget frontier, paired over S3's six seeds. If it fires, per-input thinking budgets close on this task.

Prior recorded in advance: moderate.

Result: the kill test fires -- by a hair, from on the line

Threshold	Accuracy	Mean steps	Against the fixed-budget frontier
`0.55`	`0.677`	`1.61`	`-0.096` `[-0.156, -0.036]`
`0.7`	`0.952`	`2.62`	`-0.004` `[-0.041, +0.032]`
`0.8`	`0.990`	`2.84`	`+0.011` `[-0.007, +0.030]`
`0.9`	`0.994`	`3.06`	`-0.003` `[-0.009, +0.003]`
`0.999`	`0.994`	`4.46`	`-0.006` `[-0.008, -0.004]`

(Three more thresholds in the output, same shape.) At 0.8 the adaptive model sits on the fixed-budget frontier: 0.990 at 2.84 average steps, +0.011 against what mixing budgets of two and three achieves at the same cost, with an interval that includes zero. The kill test fires because it asked for a win, and this is a tie -- but it is a tie from a starting point of -0.015 (S3) and -0.3 (S9).

Both properties, finally

Thinking steps and accuracy by number of non-zero entries, at 0.8:

Non-zero entries	1	2	3	4	5	6	7	8	9	10
steps	`2.05`	`2.17`	`2.66`	`2.93`	`2.98`	`2.99`	`3.04`	`3.19`	`3.11`	`3.33`
accuracy	`1.000`	`1.000`	`1.000`	`1.000`	`0.999`	`0.996`	`0.970`	`0.987`	`0.992`	`0.954`

The order is right -- two steps on the easiest inputs, over three on the hardest -- and the answers it stops on are right, 0.990 overall. This is the halting signal S3 and S9 were each half of. Training every step to answer was all it took.

Why it still does not win

Because a fixed budget of three steps already scores 0.999 on this task. Adaptive thinking can only beat a fixed budget where the fixed budget leaves something on the table -- where a single budget is too long for the easy inputs and too short for the hard ones. Here three steps is enough for nearly everything, so the most per-input allocation can do is spend slightly less on easy inputs, which is exactly what it does (2.84 average steps), and give back a fraction of a point on the hardest ones. The ceiling, not the method, is what stops it.

What stands, across S3, S9 and S13

Removing thinking is easy (S3: 38% for at most a point).
Allocating it needs a signal that is both ordered by difficulty and calibrated at every step. S3's learned halting was calibrated and backwards; S9's confidence was ordered and miscalibrated; S13's deep-supervised confidence is both, and it is cheap: supervise every step's answer.
With both, adaptive thinking matches the best fixed budget and does not beat it here, because on this task a fixed budget saturates. The kill test fires and the thread closes for this task.
The practical recipe that survives: train every step to answer, stop on confidence, and first measure how far a fixed budget is from the ceiling. If it is near the ceiling, there is nothing for adaptivity to win.

Limits

This task is the worst case for adaptivity, by the finding itself: a fixed budget saturates at three steps. On a task where difficulty spans a wider range of required thinking -- which real reasoning benchmarks plausibly do -- the same recipe has room to win, and that is the test worth running next, with a task designed so that no single budget is near the ceiling.
Deep supervision changes the model, not only the halting rule, so this is a different model from S3's. That is the point of the test, and it means S13 does not say S3's model could have been rescued by a different rule at inference.
Six seeds. The +0.011 could be a real small win; its interval does not rule one out.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
calibrated: A model is calibrated when its confidence matches how often it is right: answers it gives with 90% confidence should be right about 90% of the time.
deep supervision: Training a model to give a correct answer at every intermediate stage, not only at the end, so an early answer can be trusted.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.