How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

A Short Model Is a Lottery

By Joshua Bolick, Awareness Software Group ~15 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. A new model can be started from a piece of an already-trained one. Two of our own results disagreed about whether it matters that the model was made for you rather than handed round a team. One said it makes no difference. The other, published the same morning, said it costs real time. Both had tried exactly one shared model.

What we found. So we ran six. The one used in the morning's result turned out to be the worst of the six, and the difference between the best and worst was larger than the effect that result had attributed to sharing. The claim is now corrected by a banner on its own page: the penalty is real and about half the size reported.

Why it matters. The more useful finding is what changes with training. Early on, which model you happened to get matters far more than whether it was made for you, and the luck of the draw is wide enough to swallow the thing anyone was trying to measure. Train the model longer and the luck almost vanishes while the small advantage of a model made for you stays. Both earlier results were sampling the wide part of that picture, and neither conclusion was available from one try. It is worth adding that the six models had all learned their own task about equally well and were still worth very different amounts, which is the second time this week that how well a model performs turned out not to be how useful it is.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 5 additional shared donors at 4 lengths over 6 receivers, no GPU, no cost.

Program v2 Bucket R, item R11. Decisive computation: analysis/shared_donor_spread.py. Output: analysis/shared_donor_spread.json. Reproduce with python analysis/shared_donor_spread.py in about thirty minutes on a throttled laptop CPU; --reuse re-derives every endpoint in seconds.

The question

R5, published earlier the same day, found that a shared donor must be more converged than a bespoke one to be worth the same: a gap of 27.5 steps at a 160-step donor, closing to 1.7 with overlapping intervals at 800. It rests on one shared-donor seed, which R5's Limits states plainly.

**Which model you happened to get, against whether it was yours.** A new model can be started from a piece of an already-trained one. Two things change how much that helps: whether the model was made for you or handed round, and simply which one you happened to get. This separates them. Higher means the difference matters more. Early on, which model you happened to get matters far more than whether it was made for you, and the luck of the draw is wide enough to swamp the effect people were trying to measure. Train the model longer and the luck disappears almost entirely, while the small advantage of a model made for you stays roughly where it was. Two earlier experiments each tried one model and reached opposite conclusions; both were drawing from the wide part of this picture.

P8 had the same limitation and drew the opposite conclusion from it, that own and shared are indistinguishable, which is what the amortisation argument rests on. Two records, evidence of the same strength, opposite readings.

The arithmetic says it could be nothing. R5's own arm averages over six donors, one per receiver; its shared arm uses one donor six times. Both have the same expectation unless sharing itself costs something. And the shared arm's published interval is over receiver seeds only, so it cannot see donor variation at all.

Kill test, fixed before execution: the spread across the further shared donors covers R5's gap at 160 steps.

Prior recorded in advance: the kill test fires, because a 160-step donor is barely past its own transition and whether it got there should be seed-dependent. The verdict was right and the stated mechanism was wrong, see below.

Anchors, as preconditions in code: R5's control at 169.2 steps and its own-donor arm at -41.7, both re-derived from R5's committed series rather than quoted. Both reproduce exactly.

Result: the kill test fires, and R5 drew the worst donor of six

Each shared donor's head start on the receiver, by donor length.

Donor steps	Own arm	777001 (R5's)	777002	777003	777005	777007	777011	Spread
160	`-41.7`	`-14.2`	`-21.7`	`-27.5`	`-37.5`	`-27.5`	`-45.8`	`31.7`
240	`-66.7`	`-44.2`	`-60.0`	`-55.0`	`-58.3`	`-63.3`	`-66.7`	`22.5`
400	`-84.2`	`-75.8`	`-80.0`	`-75.8`	`-77.5`	`-78.3`	`-80.0`	`4.2`
800	`-90.0`	`-88.3`	`-87.5`	`-85.8`	`-82.5`	`-84.2`	`-84.2`	`5.8`

At 160 steps the six shared donors span -45.8 to -14.2, a spread of 31.7 steps, larger than the 27.5 R5 attributed to sharing. R5's donor is rank 6 of 6, the worst draw available. One shared donor, 777011 at -45.8, beat the bespoke arm outright.

The penalty is real, at less than half the size, and it does not close

With the donor as the unit of observation rather than the receiver, which is the comparison R5's design could not make:

Donor steps	Own arm	Mean over donors	Gap	R5 published
160	`-41.7`	`-29.0` `[-40.8, -17.2]`	`12.6`	`27.5`
240	`-66.7`	`-57.9` `[-66.2, -49.7]`	`8.8`	`22.5`
400	`-84.2`	`-77.9` `[-79.9, -75.9]`	`6.3`	`8.4`
800	`-90.0`	`-85.4` `[-87.7, -83.1]`	`4.6`	`1.7`

Two corrections in opposite directions. At short lengths R5 overstated the penalty by more than double. At 800 steps R5 reported 1.7 with overlapping intervals and called the arms equivalent; with six donors the gap is 4.6 and the own arm sits outside the donor-level interval.

So a bespoke donor really is worth a few steps more than a shared one, at every length tested, and the size of that advantage is roughly constant rather than converging away. R5's headline description, a penalty that shrinks as the donor converges, was reading the collapse of the variance as a collapse of the mean.

The finding worth keeping: the lottery closes, the penalty does not

The spread falls by a factor of seven while the gap falls by less than three:

Donor steps	Spread across donors	Own-minus-shared gap
160	`31.7`	`12.6`
240	`22.5`	`8.8`
400	`4.2`	`6.3`
800	`5.8`	`4.6`

Below about 240 steps, which donor you happen to have matters more than whether it is yours. Above it, donor identity stops mattering almost entirely and a small bespoke advantage remains.

This is the unifying explanation for P8 and R5 disagreeing. Each ran exactly one shared donor and each drew from a distribution this wide. P8 drew one that matched its bespoke arm and concluded they were equivalent; R5 drew the worst of six and concluded sharing costs 27.5 steps. Neither conclusion was available from one draw, and both records say the limitation out loud without either noticing it could produce the other's answer.

The prior was right and its stated mechanism was not

The prior predicted a large spread because "a 160-step donor is barely past its own transition and whether it got there at all is seed-dependent". The donors are not seed-dependent in that way. Their own accuracies at 160 steps span 0.2995 to 0.3267, a range of 0.027: they all learned about equally well, and they were still worth between -14.2 and -45.8 steps to a receiver.

Being equally good is not being equally useful, which is R6's delayed-copy-2 row arriving by a different route on the same day.

A weak association does show up in the direction one would expect, and it is six points, so it is an indication and not a measurement: rho(accuracy, worth) is -0.551, -0.486, -0.412 at 160, 240 and 400 steps, and +0.580 at 800 where the spread has collapsed to 5.8 steps and the sign flip is what noise looks like.

A provenance repair, made here rather than left open

R5 printed its shared donor's own accuracy per length and stored only the effects, so the number that would say whether it was an unlucky model or an unlucky draw was missing from R5's output. Recomputing is deterministic: same seed, same config, same lengths, about thirty seconds, and gives 0.3065 at 160 steps, fourth of six. So R5's donor was of ordinary quality and unusually unhelpful, which is the point.

None of R5's effects were retrained. Its controls, own-donor arm and original shared-donor effects are read from its committed JSON, so every comparison above is against the numbers R5 published.

What stands

R5's shared-donor penalty is about half real. 12.6 steps at 160, not 27.5, and R5 drew the worst of six donors.
It does not close at convergence. 4.6 steps at 800 with the own arm outside the interval, where R5 reported 1.7 and equivalence.
What closes is the variance, 31.7 to 4.2. A short donor is a lottery and a converged one is not.
P8 and R5 disagreed because each had one draw from a distribution wider than the effect either claimed. A banner is added to R5; P8's own-versus-shared claim is on a different task and is not withdrawn, but its evidence is one draw and that is now quantified.
Donor accuracy and donor usefulness are different quantities, again.

Limits

Six donors is six donors. The intervals over donors are t-intervals on n = 6, which is the same small n this programme uses everywhere, applied to a unit it has not used before.
One task, one width (48), one learning rate, and the receivers are R5's six seeds throughout, so the receiver side is held fixed rather than resampled.
Four lengths, not seven. R5's two shortest rungs (40, 80) are omitted: both delay the receiver in R5's table, so there is no head start there to have a spread about.
The rho column is six points. It is reported because leaving it out after computing it would be worse, and it should not be cited as a measurement of anything.
Nothing here prices a donor. The break-even arithmetic in R5 and P8 is untouched; what changes is how much confidence one shared donor buys.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
variance: How spread out a set of numbers is. Rising variance in a signal means it is becoming more erratic.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.