How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Two Orderings, Untied

By Joshua Bolick, Awareness Software Group ~13 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: Which parts of a model actually matter? – The parts that move most are not the parts that matter, and no single component is required -- the model routes around every freeze. What a part is worth shows up only when you remove it.

In plain English

What we asked. When you start a new model from a piece of an old one, two things about the old model's task might tell you how much it will help: how hard that task was, and how close it is to the job you actually want done. An earlier experiment of ours could not tell those two apart, because in the four tasks it used the easiest one also happened to be the least similar. It said so at the time.

What we found. Adding five more tasks separates them. Once separated, how hard the old task was almost stops predicting anything, while how close it is gets stronger. And when we compare pairs of candidates that are equally close to the job, being trained on a harder task does not make a candidate better: it wins about a third of the time, which is what no effect looks like.

Why it matters. So the practical advice is the simple one. Choose what to build on by what its task has in common with yours, not by how demanding that task was. Two smaller things are worth noting. Four measurements from the earlier experiment came back identical here, on runs trained months apart by different code, which is the best evidence in the record that both are measuring the same thing. And the hint that easier is actually better rests on comparisons that share candidates with each other, so we report it and do not claim it.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU. The only new training is nine controls, six seeds each; everything else is read from R6's committed grid.

Program v2 Bucket R, item R8. Decisive computation: analysis/difficulty_versus_relatedness.py. Output: analysis/difficulty_versus_relatedness.json. Reproduce with python analysis/difficulty_versus_relatedness.py in about fifteen minutes on a throttled laptop CPU; --reuse re-derives every endpoint from the saved controls.

The question

R3 answered its kill test and could not settle its question. Its four tasks were chosen by Q14 for relatedness coverage, and in them the easiest task is also the least related, so difficulty and relatedness are entangled. R3 reported rho -0.457 for difficulty against rho -0.754 for relatedness and said, in as many words, that the -0.457 was a confound rather than a finding.

Which matters more: how hard the old task was, or how close it is. When you start a new model from a piece of an old one, two things about the old model's task might predict how much it helps: how hard that task was, and how close it is to the new one. An earlier experiment could not tell them apart, because in its four tasks the easiest was also the least similar. This adds five more tasks, which separates them. Once the two are separated, how hard the old task was almost stops predicting anything, while how close it is gets stronger. The earlier experiment had said its own difficulty number was probably an artefact of the tasks it happened to use; this measures that rather than arguing it. Pick a candidate model by what its task has in common with yours, not by how demanding it was.

Kill test, fixed before execution: with relatedness held constant, donor difficulty does not order the transfer effects.

Prior recorded in advance: it does not order them, which would make R3's answer a general one rather than a fact about four tasks.

Anchor: Q4's same-task dispatch cell must reproduce at -90.0 from R6's committed series, as a precondition in code. It does, exactly.

The design is a band, not a ladder, and that is a deliberate substitution

R8 was written asking for a designed ladder that varies difficulty while holding relatedness constant. Building one runs straight into the reason R3 could not settle this: on these families almost every knob that makes a donor's task harder also changes how much the donor knows about the receiver's task.

So this runs the cheaper precursor: measure both quantities on R6's nine donor configurations and use the bands where relatedness is already matched. If the two cannot be untied even there, a designed ladder is not merely expensive but impossible in this family, and that is the answer rather than a delay. They can be untied: R6's grid supplies fourteen matched pairs, one band three donors wide within 0.012 of relatedness.

Difficulty, and a free anchor nobody planned

Each task's own control reaching its own level, R3's definition and R3's computation.

Task	Steps	Task	Steps
`running-count`	`37.5`	`noisy-copy-35`	`100.0`
`delayed-copy-2`	`59.2`	`dispatch-4lag`	`103.3`
`delayed-copy-4`	`86.7`	`dispatch-8lag`	`122.5`
`noisy-copy-15`	`91.7`	`delayed-copy-7`	`127.5`
		`modular-sum`	`485.8`

All four of R3's published values reproduce exactly, 37.5, 86.7, 91.7, 122.5, on runs trained months apart by a different script. That was not planned as an anchor and it is the strongest evidence in this record that the two are measuring the same quantity.

Result: the kill test fires

On the wider grid, R3's confound dilutes exactly as R3 predicted it would.

Ordering	R3, four tasks	Here, sixteen cross-task cells
donor task difficulty	`-0.457`	`-0.221`
donor relatedness	`-0.754`	`-0.882`

Difficulty's apparent signal halves while relatedness strengthens. R3 called the -0.457 a confound rather than a finding; on a grid where the entanglement is looser, it behaves like one.

And with relatedness held constant, difficulty does not order the effects at all.

Matching threshold	Pairs	Harder donor wins
`0.03`	`9`	`2` (`22.2%`)
`0.05` (fixed in advance)	`14`	`5` (`35.7%`)
`0.08`	`18`	`8` (`44.4%`)

The harder donor wins about a third of the time, and the fraction climbs toward 50% as the threshold loosens and relatedness variation leaks back in, which is what a confounded signal looks like when you stop controlling it.

The pairs where both donors actually help

The cleanest subset, because a comparison between two harmful donors only asks which harms less.

Among the eight matched pairs where both donors help, the harder one wins 1. At matched relatedness, the easier donor was better seven times out of eight.

This is suggestive and it is not eight independent trials. With nine donors the pairs share members heavily, and delayed-copy-7, the second-hardest task and a mediocre donor, appears in five of the fourteen pairs. A large part of "harder donors are worse at matched relatedness" is one donor's story, so the direction is reported and not claimed. What does not depend on it is the kill test: difficulty does not order the effects at any of the three thresholds.

What stands

The kill test fires. With relatedness held constant, donor difficulty does not order transfer.
R3's answer generalises, and R3's own caution about its -0.457 is now measured rather than argued: on a wider grid it falls to -0.221 while relatedness rises to -0.882.
Difficulty and relatedness can be untied in this family, which R8 was written doubting. A designed ladder is possible; it is also no longer necessary for this question.
R3's four difficulty values reproduce exactly on independently trained runs.
The direction at matched relatedness may be the opposite of the hypothesis, easier donors slightly better, and rests on non-independent pairs dominated by one donor. Not claimed.

Limits

Nine tasks, two receivers, sixteen cross-task cells. Wider than R3's twelve, still small.
The matched pairs are not independent. Above; this is the main reason the reverse direction is reported rather than concluded.
modular-sum is an outlier by a factor of four (485.8 steps) and R6 recorded that it never converged at 800 donor steps. Its pairs are with running-count and delayed-copy-2, where nothing helps, so they contribute to the count without being informative about useful donors. They are left in rather than dropped.
"Difficulty" is time-to-own-level at one width and one learning rate, R3's operationalisation and R3's limit. A task's plateau might order the tasks differently.
Relatedness and worth are R6's committed numbers, so this record inherits R6's endpoint (its pooled level) for worth. The anchor alone uses L7's criterion, exactly as R6 does.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

confound: A second explanation you did not control for. If bigger models both learn faster and score higher, then 'fast learners score higher' may be entirely about size and not about speed.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
plateau: A stretch of training where the model is not visibly improving. Often, though not always, followed by a sudden jump.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.