How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Which Donors You Count

By Joshua Bolick, Awareness Software Group ~13 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. A new model can start from a piece of an older one, and a free one-pass check tells helpful candidates from harmful ones. An earlier experiment of ours said the check could reject the bad candidates but not rank the good ones. We tested three other free scores to see whether any ranks the good ones better.

What we found. None did. And the free check itself ranked the good candidates well, better than we had said. The earlier claim came from averaging its scores across two different target jobs, where the scores sit on different scales; looked at one job at a time, it orders them sensibly. That earlier page now carries a correction note. Then a twist: if you also leave out candidates from the same family of tasks as the target, which is the fairer test, how fast each candidate had learned its own task ranked them far better.

Why it matters. Because we chose that fairer pool after seeing the results, it counts as a lead, not a finding, and we have written it down as the next test on fresh runs. The lesson worth copying: when two reasonable definitions of the comparison group give different winners, the definition is doing the deciding, and it needs fixing in advance.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, R6's six seeds and nine donors retrained exactly; the receiver side is R6's committed series. The kill test was written on 2026-09-04 and its operational definitions (signs, band, unit) committed in the pilot (a72c374) before any run.

Program v2 Bucket R, item R13. Decisive computation: analysis/donor_ranking_rule.py. Output: analysis/donor_ranking_rule.json. Reproduce with python analysis/donor_ranking_rule.py (about twenty-five minutes on a throttled laptop CPU); --reuse re-derives every endpoint from the saved scores.

The question

R6 found that Q14's free screen -- a donor's zero-shot accuracy on the receiver's task -- separates helpful donors from harmful ones perfectly, and concluded it is a rejection rule, not a ranking rule: "the two most valuable donors are the two the screen ranks lowest among the helpful ones". So: what should rank the donors the screen keeps?

**Ranking the useful donor models: which free score, and which donors count.** When a new model can start from a piece of an old one, a free check sorts the helpful candidates from the harmful. This asks what best ranks the helpful ones. Each score is tested twice: leaving out only a candidate trained on the very same task (blue), and leaving out the whole family of similar tasks (green). 'Speed' is how fast the candidate learned its own task; 'gates' is a six-number summary of its internals. With only the same task left out, the free check ranks best and nothing beats it. Leave out the whole family of similar tasks and how fast each candidate learned its own task ranks far better. Which candidates you count decided the answer, so the second result is a lead to test properly, not a conclusion.

Three free candidates, with the direction each is expected to point fixed before execution: the donor's zero-shot loss (lower better; M6 found loss leads accuracy, and accuracy saturates), the donor's own transition time on its own task (sooner better), and the distance between the donor's six gate-bias means and the receiver's own-task donor's (closer better; P7's recipe).

Unit (R15's rule): each seed and receiver separately -- donors are trained per seed -- scored as agreement, the sign-corrected rank correlation with measured worth across the helpful band: the donors, other than the receiver's own task, whose mean worth in R6 is negative.

Kill test, fixed before execution: none of the three has mean agreement above zero-shot accuracy's, paired over seed-and-receiver units, with an interval excluding zero.

Prior recorded in advance: moderate for the loss version.

Anchor, in code, passed exactly: every one of the 54 retrained donors reproduces R6's committed final accuracy, and every zero-shot score R6's screen value, with a largest difference of 0.

Result: the kill test fires

Twelve seed-and-receiver units. Agreement with measured worth across the helpful band:

Score	Agreement	Minus zero-shot accuracy, paired
zero-shot accuracy (the screen)	`+0.685` `[+0.490, +0.879]`
zero-shot loss	`+0.195` `[-0.162, +0.552]`	`-0.490` `[-0.695, -0.284]`
donor transition time	`+0.569` `[+0.334, +0.803]`	`-0.116` `[-0.253, +0.021]`
gate-bias distance	`+0.174` `[-0.092, +0.440]`	`-0.511` `[-0.706, -0.316]`

Nothing beats the screen. Loss and the gate summary are clearly worse; transition time is close and not better. My prior on loss was wrong: accuracy did not saturate within the helpful band the way it does across the whole pool.

The screen is not backwards -- and R6 did not say it within a receiver

Within each receiver, the screen orders the helpful band well: +0.685. That reads as the opposite of R6's "close to inverted", and it is worth being exact about why it is not a contradiction of R6's data. R6's table has one row per donor, pooled over both receivers, and averages screen scores measured on two different tasks -- zero-shot accuracy on delayed-copy runs 0.5-0.98 among the helpful donors, on dispatch it runs 0.1-0.43. Pooled, a donor's average screen score mostly says which receiver it helps, not how much. Per receiver, the ordering holds, which is what the +0.685 above measures.

So R6's conclusion that the screen is "a rejection rule, not a ranking rule" rests on a cross-receiver average. R6 carries a qualification banner. Its pool-size curve -- the screen's advantage flat past three candidates -- is computed per receiver and is not affected.

And the band decides the runner-up

R6 ran its pool-size analysis twice and called the own-family-removed version the honest one, because a near-identical donor makes any rule look good. R13's band, as fixed in advance, removes only the receiver's own task. Recomputed with R6's convention -- post hoc, not the kill test:

Score	Agreement, own family removed	Minus accuracy
zero-shot accuracy	`+0.429` `[+0.003, +0.854]`
donor transition time	`+0.862` `[+0.791, +0.933]`	`+0.434` `[+0.067, +0.801]`
zero-shot loss	`+0.183`
gate-bias distance	`+0.299`

Under R6's own convention, the donor's own transition time ranks the helpful band clearly better than the screen. With the same-family donors in, the screen wins because a same-family donor is both the most helpful and the most obviously related, which is exactly what R6 warned makes ranking trivial.

This does not overturn the kill test, which was fixed with the other band. It is a post hoc finding from a choice made after seeing the data, and this programme's rules say such a finding is a question, not an answer. It generates R17: the same comparison, band fixed in advance as R6's own-family removal, on fresh seeds.

One caution for R17, from R8: difficulty did not order transfer once relatedness was matched. A donor's transition time is close to its task's difficulty, so if it ranks donors here it may be doing so as a proxy for relatedness. That would still make it a free ranking signal; it would not make difficulty the mechanism.

What stands

The kill test fires. No candidate beats zero-shot accuracy with the band fixed in advance; loss and gate distance are clearly worse.
The screen ranks well within a receiver (+0.685); R6's "close to inverted" came from averaging screen scores across two receivers. R6 qualified by banner.
Which donors count decides the runner-up. With the receiver's own family removed -- R6's own honest convention -- transition time beats the screen (+0.434 [+0.067, +0.801]), post hoc.
Generates R17, the preregistered version of that comparison.

Limits

Two receivers and four or five helpful donors each, so each unit's rank correlation rests on few points; the intervals are over twelve units, and R6's seeds are the only ones used.
The helpful band is defined from R6's mean worth, which is computed from the same series the agreement is scored against. That is how R13 was specified, and it is why the band definition matters.
R8's caution on what transition time is a proxy for, above.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
post hoc: Worked out after the fact, rather than decided in advance. We report such checks separately and never let them decide a result, because it is far too easy to find a pattern once you already know the answer.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.