How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Where the Shortcut Stops Paying

By Joshua Bolick, Awareness Software Group ~12 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Part of a bigger question: How do we know our own results are real? – Repeatedly, the control rather than the measurement decided the result -- and several striking findings dissolved when the right comparison was finally run.

In plain English

What we asked. We have exactly one shortcut that saves anything: partway through training, switch to a cheaper setting that processes less data per step. The page before this one established that whether it helps depends on the size of the model, and left the boundary somewhere between two sizes four times apart. This finds it.

What we found. At four model sizes the shortcut saves 21%, 20%, 13%, and then costs 42%. It pays below about a hundred units of width and not above. Our test asked whether the decline is steady, and technically it is, so the shortcut is a small-model trick that fades away as models grow. But the shape is worth more than the verdict: the first three sizes sit within eight points of one another and the fourth falls fifty-five. That is a cliff, not a fade.

Why it matters. There is a candidate explanation sitting in the control runs, and it is checkable. Simply making the model bigger had itself been buying less and less: going from the first size to the second made training three times cheaper, the next step 1.6 times, and the last step only nine per cent, despite four times the parameters. By the largest size the task has almost stopped rewarding extra capacity, so there is very little left for a cheaper setting to recover while its downside is unchanged. On four points that is a shape rather than a proof, but it makes a prediction: the cliff sits wherever extra size stops paying, not at any fixed width. Separately, and for the fifth time in this programme, timing the switch to each individual run bought nothing over using one fixed moment for all of them - and at the largest size it was actively worse.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 60 training runs, no GPU, no cost.

Program v2 Bucket Q, item Q12. Decisive computation: analysis/cut_width_sweep.py. Output: analysis/cut_width_sweep.json. Reproduce with python analysis/cut_width_sweep.py; --reuse re-derives every endpoint without retraining.

The question

Q11 left a sign change bracketed and unlocated: cutting the batch 4x saves +19.9% of samples-to-target at width 48 and costs -42.0% at width 192, at the same multiple of each run's own transition on the same task. This is the only efficiency manoeuvre in this archive with a positive region at all, so where its boundary sits is the most useful thing left to know about it.

**The shortcut stops paying exactly where extra model size stops paying.** This is the one shortcut we have found that saves anything. It works on small models and we did not know where it stops. The solid line is what it saves at each size. The dashed line is a different quantity on the same axis: how much cheaper simply making the model bigger had already made training, at each step up in size. The shortcut pays below about a hundred units of width and not above it, and the drop is a cliff rather than a slope: three sizes within eight points of each other, then a fall of fifty-five. The dashed line suggests why. By the largest size, making the model bigger has almost stopped helping - four times the parameters buys nine per cent - so there is very little left for a cheaper setting to recover, while its downside is unchanged. On four points that is a shape rather than a proof, but it predicts the cliff sits wherever extra size stops paying, not at any fixed width.

Four widths at lag 4, control against an oracle arm at x0.8 of each run's own transition, Q11's best-performing before-arm, plus a fixed-schedule arm beside it.

Kill test, fixed before execution: the saving is monotone in width. Then the manoeuvre is a small-model effect that extrapolates cleanly to nothing. If it is not monotone, width is standing in for something else and this is a new question rather than a closed one.

The anchors are exact

All four reproduce Q11 to the digit: same seeds, same code path:

	Here	Q11
width 48 control	`10,944` `[9,772, 12,116]`	`10,944` `[9,772, 12,116]`
width 48 `x0.8`	`8,771` `[7,586, 9,957]`	`8,771` `[7,586, 9,957]`
width 192 control	`6,080` `[5,799, 6,361]`	`6,080` `[5,799, 6,361]`
width 192 `x0.8`	`8,634` `[7,335, 9,933]`	`8,634` `[7,335, 9,933]`

Result: the kill test fires, and the shape is more interesting than the verdict

Width	Transition	Control samples to 0.9	Oracle `x0.8`	Fixed step
24	`138`	`36,080` `[20,918, 51,242]`	`+21.1%`	`+20.4%`
48	`88`	`10,944` `[9,772, 12,116]`	`+19.9%`	`+19.3%`
96	`56`	`6,656` `[6,324, 6,988]`	`+12.9%`	`+12.7%`
192	`39`	`6,080` `[5,799, 6,361]`	`-42.0%`	`-35.0%`

Monotone, so the kill test fires, and the manoeuvre is a small-model effect. But monotone is doing less work here than it sounds. The first three widths sit within 8 points of each other, +21.1, +19.9, +12.9, and the fourth falls 55 points. That is a cliff, not a fade, and a "small-model effect that extrapolates to nothing" is a fair description of the verdict and a poor description of the curve.

A candidate mechanism, visible in the control column

The control's own cost stops improving at exactly the width where the cut stops paying:

Step	Control samples to target	Improvement
24 → 48	`36,080` → `10,944`	`3.30x` cheaper
48 → 96	`10,944` → `6,656`	`1.64x` cheaper
96 → 192	`6,656` → `6,080`	`1.09x` cheaper

By width 192 the control is within 9% of the width-96 control despite four times the parameters. The task is saturating in samples-to-target, so there is very little headroom left for a cheaper regime to exploit, and the cheap batch's own cost is unchanged. On four points this is a coincidence of shape rather than a demonstrated cause, but it is checkable and it predicts the cliff sits wherever the marginal return to width collapses, not at any particular width.

The fixed-schedule arm, for the fifth time

Oracle against one population number for every seed: +0.6, +0.6, +0.1 points at widths 24, 48 and 96, and -7.0 at width 192, where the fixed arm is the better of the two. Per-run timing buys nothing anywhere here, and at the width where the manoeuvre fails it actively hurts. That is K2's finding, Q1's, Q5's, Q11's and now this one.

What this settles and what it does not

The batch cut pays below roughly width 100 on this task and not above it. That is the boundary Q11 bracketed, now located to a factor of two.
Per-run timing is worthless across the whole range, so the one instrument this manoeuvre was built around (F2's decode probe) has nothing left to contribute even where the manoeuvre works. Combined with Q9's pricing, the probe costs 21.9x the training it watches at width 48, the economics stay closed.
The cliff wants explaining, and the control column offers a candidate that a difficulty-matched ladder would test directly.

Limits

This is a fixed-task width sweep, which is P11's confound by construction: deliberately, because the axis under test is width against a fixed task. What "width" means here is therefore spare capacity: the transition falls 138 → 39 across the sweep, so the width-192 model is enormously oversized for lag 4. Whether the cliff survives a ladder where difficulty rises with width is untested and is the obvious follow-up (Q13).
The width-24 control interval is very wide (20,918 to 51,242), because a narrow model's time-to-target varies a great deal across seeds. The +21.1% there is the least reliable number in the table.
Four widths locate the crossing to a factor of two and nothing here says the curve is smooth between 96 and 192. A point at 128 or 144 would say whether the cliff is a step or a steep slope.
One task, one cut ratio, one multiple, one target. All inherited from J7 and Q11 and fixed in advance.
The oracle arm needs a control run per seed and is not deployable. The fixed arm is what a practitioner would use, and on this evidence it is what they should use.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

confound: A second explanation you did not control for. If bigger models both learn faster and score higher, then 'fast learners score higher' may be entirely about size and not about speed.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
parameters: The adjustable numbers inside a model. Training is the process of setting them. Model size is usually quoted as a count of these.
probe: A small separate model trained to read information out of a bigger model's internals, used as a measuring instrument rather than as a product.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
slope: How steeply one quantity changes as another does. A slope of one between a warning and the event it predicts means the warning shifts exactly in step with the event.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.