How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

A Free Check Saturates at Three

By Joshua Bolick, Awareness Software Group ~15 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. A new model can be started from a piece of an already-trained one. Sometimes that saves a lot of training and sometimes it costs more than starting from scratch, and which one you get depends on which trained model you picked. We have a check that guesses in advance, and it costs one pass of the data, which is nothing.

What we found. We already knew the check works when you have three candidates to choose between. This asked what happens when you have more. The answer is that almost nothing happens. The check is worth about twenty steps of training and it gets essentially all of that from the first three candidates. Going from three to six buys nothing measurable.

Why it matters. Meanwhile the best choice available in the pile keeps getting better, because a bigger pile contains better options. So the gap between what the check picks and what was actually there widens, from about nine percent of the available benefit to about twenty one percent. We wrote that prediction down before running it, along with the reason: the check is very good at telling a helpful model from a harmful one and only mediocre at ranking the helpful ones against each other. The practical version: use it to throw candidates out, not to pick a winner. One row makes the point on its own. A model that had learned its own task perfectly still cost a new model time, and the check caught it. How well a model performs is not the same question as whether it is useful to build on.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 9 donor configurations by 2 receivers over 6 seeds, no GPU, no cost.

Program v2 Bucket R, item R6. Decisive computation: analysis/screening_pool_size.py. Output: analysis/screening_pool_size.json. Reproduce with python analysis/screening_pool_size.py in about forty minutes on a throttled laptop CPU; --reuse re-derives every endpoint from the saved series in seconds.

The question

R2 found that Q14's one-forward-pass screen does not improve P8's break-even, it creates it: a donor drawn without looking is a net loss, while the best-scoring donor of the pool saves real steps.

**What a free check is worth, as the pile of candidates grows.** A new model can be started from a piece of an already-trained one, which sometimes helps a lot and sometimes costs more than starting from scratch. A check costing one pass of the data guesses which. This shows what three ways of choosing are worth as the number of candidates grows: taking one at random, taking the one the check likes, and taking the one that actually turns out best. Lower is better; zero means no difference from starting fresh. Taking one without looking is flat, and slightly worse than not reusing anything at all. The free check is worth about twenty steps and it gets all of that from the first three candidates: after that its line stops improving while the best available choice keeps getting better, so the widening gap is what the check leaves on the table. It is good at rejecting the bad candidates and not good at ranking the good ones, so a bigger pile is worth very little to it.

R2 is an existence result on a pool of three, and says so. The practical question is the curve. The donor pool is widened here to nine configurations spanning the relatedness range, so each receiver chooses from eight after its own configuration is removed.

Kill test, fixed before execution: the screened advantage does not increase between k = 3, R2's pool, and k = 8, the largest tested.

Prior recorded in advance: the advantage does increase, because a maximum over more candidates is better almost by definition, so the kill test as written is likely to survive on a technicality. The prediction worth testing is that the fraction of the oracle's gain the screen captures falls as the pool grows, Q14's measure separates helpful from harmful perfectly but orders them only at rho = -0.754, and separation is what wins at small k while ordering is what wins at large k.

Anchor: Q4's same-task dispatch cell reproduces at -90.0, exactly.

Two columns, and they disagree

Both were in the committed script before execution (27edd47). "Blind" is the expectation over the pool rather than a simulated draw, so it is flat in k by construction and is a reference line, not a competitor.

Own configuration removed: pool of 8

`k`	Pick blind	Pick by the screen	Pick the best there is	Screen gets
1	`-0.1`	`-0.1`	`-0.1`	,
2	`-0.1`	`-20.2`	`-21.0`	`96.0%`
3	`-0.1`	`-26.2` `[-30.0, -22.5]`	`-27.8`	`94.4%`
4	`-0.1`	`-30.0`	`-32.0`	`93.7%`
6	`-0.1`	`-36.0`	`-37.7`	`95.6%`
8	`-0.1` `[-6.4, +6.3]`	`-41.7` `[-47.9, -35.4]`	`-41.7`	`100.0%`

The kill test does not fire. The advantage grows from -26.2 to -41.6 steps, and the screen captures everything available by k = 8.

Own family removed: pool of 6

`k`	Pick blind	Pick by the screen	Pick the best there is	Screen gets
1	`+4.3`	`+4.3`	`+4.3`	,
2	`+4.3`	`-16.1`	`-17.0`	`95.5%`
3	`+4.3`	`-20.1` `[-28.3, -12.0]`	`-22.4`	`91.4%`
4	`+4.3`	`-21.2`	`-25.0`	`87.1%`
5	`+4.3`	`-21.1`	`-26.3`	`82.9%`
6	`+4.3` `[-4.1, +12.8]`	`-20.5` `[-31.3, -9.8]`	`-27.0`	`79.3%`

The screened advantage is flat from k = 3 onward, -24.5 to -24.9 paired against blind, a difference well inside half an interval: while the best available choice keeps improving, -22.4 to -27.0. The screen's share falls from 91.4% to 79.3%.

What the two columns mean

The headline column is confounded by near-identical donors. With only the receiver's own configuration removed, a delayed-copy-4 receiver still gets delayed-copy-2 and delayed-copy-7 in its pool, and a dispatch-8lag receiver still gets dispatch-4lag. A near-identical donor makes the choice trivial for any rule, which is why the screen reaches 100%. That is a fact about the pool, not about the screen.

The family-removed column is the honest test, and the recorded prior holds in it. Screening buys its whole advantage from the first three candidates. Everything after that accrues to the oracle and not to the screen.

So R2's result should be read as "the screen avoids disasters", not "the screen finds the best donor", which is what the prior predicted and what Q14's own numbers imply: perfect separation, mediocre ordering.

The per-donor table shows the mechanism

Descriptive, pooled across both receivers, so it includes same-family cells and is an illustration rather than the evidence. The evidence is the curve above.

Donor	Learned its own task	Worth to a receiver	Screen score
`delayed-copy-4`	`0.991`	`-28.8`	`0.5604`
`noisy-copy-15`	`0.842`	`-24.2`	`0.5590`
`noisy-copy-35`	`0.636`	`-17.5`	`0.5509`
`dispatch-4lag`	`0.816`	`-35.0`	`0.5375`
`dispatch-8lag`	`0.529`	`-33.8`	`0.5234`
`delayed-copy-7`	`0.839`	`-13.8`	`0.3856`
`delayed-copy-2`	`1.000`	`+9.2`	`0.0570`
`modular-sum`	`0.227`	`+15.4`	`0.0306`
`running-count`	`1.000`	`+82.1`	`0.0062`

The separation is total and the ordering is close to inverted. Every harmful donor scores below 0.06 and every helpful one above 0.38, with nothing between: Q14's gap, reproduced on a wider pool. But the two most valuable donors are the two the screen ranks lowest among the helpful ones.

delayed-copy-2 is the row worth staring at. It learned its own task perfectly, 1.000, and it still costs a receiver +9.2 steps. Mastery of a task is not what makes a donor useful, and the screen catches it for one forward pass where the donor's own accuracy would not have.

modular-sum never converged (0.227 mean, as low as 0.063 on one seed), so that pool member is partly "an untrained donor" rather than "a donor from task X". It is left in: a real pool contains duds, the screen rejects it correctly at 0.0306, and removing the one member a method handles well after seeing that it handles it well is how a result gets better than it is.

The anchor fired first, and that is reported rather than tidied away

The first execution read -55.0 against Q4's -90.0 and the script refused to interpret anything, saving the raw runs and exiting 2.

It was my error and not Q4's, and it is the mistake Q14's own docstring documents. I read Q4's number at this pilot's level: the pooled floor-to-plateau midpoint, which the pool analysis needs because five of the nine tasks here have no published ceiling. Q4's record says in as many words that the effect widens with the level it is read at, so a number for it is a number at a level. Read under L7's task criterion, as Q4 and R5 read it, it reproduces at -90.0 exactly.

Nothing was retrained. The raw runs were written to disk before any summary ran, so --reuse re-derived every endpoint in seconds. That is the standing rule about saving first paying for itself in the same session it was followed.

What stands

A free screen buys its whole advantage from about three candidates. Beyond that, on a pool with near-identical donors removed, a wider pool is worth nothing to it.
The screen's share of what is available falls as the pool grows, 91.4% to 79.3%, exactly as the recorded prior predicted and for the reason it gave.
The kill test as written does not fire, and the prior said it would survive on a technicality. Reported as stated rather than rewritten.
A donor that has mastered its own task can still cost a receiver steps, and the screen catches that case.
Screening is a rejection rule, not a ranking rule. Anyone with more than a handful of candidates and a desire to pick the best one needs a different instrument.

Limits

Two receivers. Both are published families, but the curve is an average over two.
The pool is nine configurations of five families, so "pool size" here varies the number of candidates, not the number of independent task families. A pool of eight genuinely unrelated tasks might behave differently, and the family-removed column is the closest thing here to that.
modular-sum did not converge, above.
The blind arm is an expectation, not a draw. It has no sampling noise, which makes the paired comparison tighter than a simulated draw would be. It is the right reference for "what is a donor worth if you do not look", and it is not a simulation of picking one.
One width (48), one learning rate, six seeds, inherited from Q14.
k is a pool size, not a cost. Acquiring candidates costs training runs, which is what P8 measured and this does not.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
plateau: A stretch of training where the model is not visibly improving. Often, though not always, followed by a sudden jump.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.