How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Two Ladders That Had to Agree

By Joshua Bolick, Awareness Software Group ~13 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. One of our results says that reusing a trained model to speed up a later one works best when the first model was trained for a particular amount of time. Finding that amount meant trying seven options, which costs more than the whole technique saves. So the obvious next question is whether it can simply be guessed from something we already know.

What we found. A single rule guessed it exactly, on both of the experiments we have. That looks like a good day. It is not, and the reason is worth more than the answer would have been.

Why it matters. The second experiment's seven options were chosen by scaling the first experiment's by exactly the quantity the rule uses. So the rule and a much duller explanation, that the answer just happens to sit in the same slot on both lists, make predictions four percent of a slot apart. Nothing we measured could have told them apart, and the rule could not have failed. Neither decision was a mistake: the scaling was the right way to make the two experiments comparable, and it was made a day earlier for a different reason. The general lesson is that scaling one experiment by a quantity to line it up with another quietly makes that quantity impossible to test on the pair. We have written down the third experiment that would settle it, and what it would cost.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. No new training. Re-analyses two committed ladders.

Program v2 Bucket R, item R9. Decisive computation: analysis/donor_length_optimum.py. Output: analysis/donor_length_optimum.json. Reproduce with python analysis/donor_length_optimum.py in under a second.

The question

R5 found the break-even against donor length is U-shaped, with an optimum in the middle that P8's ladder could not see. Finding that optimum cost a seven-rung sweep, which is more compute than the technique saves. The shape is only useful if the optimum can be located cheaply.

**Two experiments agreed, and the way they were built is why.** Reusing a trained model to speed up a later one works best when the first model was trained for a particular amount of time. This asks whether that amount can be guessed from something already known, instead of found by trying every option. The guess is a single multiple of how long the task takes to learn from scratch, shown against what each of two experiments actually measured. The one rule matches both experiments almost exactly, which looks like a result and is not one. The second experiment's list of options was built by scaling the first one's by exactly the quantity the rule uses, so the rule and a far duller explanation, that the answer is simply in the same slot on both lists, make predictions four percent of a slot apart. Nothing measured here could tell them apart. A third experiment with an unscaled list would.

Kill test, fixed before execution: neither candidate predicts the optimum to within one rung across the two tasks now measured.

Prior recorded in advance, in docs/backlog.yml at commit 27edd47, written before any of these numbers were computed:

the donor's own transition predicts it, because the optimum on both tasks sits at roughly 2.4x the receiver control's transition and that is the one number the scaling used

The form of that prior was right and the multiplier in it was wrong. 2.4x came from reading R5's shared-donor column, which is the only one whose economics R5 quotes. The comparable column is the own-donor one, because P8's whole ladder is per-receiver donors and P8 ran its shared donor at one length only.

The predictor hits both optima exactly

Ladder	Control transition	Optimum	Rung index	Break-even	As a multiple
`delayed-copy` (P8)	`86.7`	`120`	`4`	`4.80`	`1.38x`
`dispatch-copy-8lag` (R5, own donor)	`169.2`	`240`	`4`	`3.60`	`1.42x`
`dispatch-copy-8lag` (R5, shared donor)	`169.2`	`400`	`5`	`5.30`	`2.36x`

Fitting one constant across both own-donor ladders gives c = 1.40, and it predicts 120 and 240: both optima, exactly, off by zero rungs. The kill test does not fire.

And the test cannot tell that apart from a much duller explanation

R5 scaled P8's ladder by the ratio of the two tasks' control transitions. That was the right decision for R5: copying P8's rungs unscaled would have tested whether P8's step counts transfer, which nobody claimed. It is also the decision that makes this test uninformative.

Quantity	Value
R5's ladder over P8's, rung for rung	`2.000`
The two control transitions	`1.952`
The two hypotheses therefore disagree by	`2.5%`
At the optimum, in steps	`5.9`
In rungs	`0.037`

"The optimum is 1.40x the control transition" and "the optimum is at rung index 4" make predictions 0.037 of a rung apart on these ladders. Both optima are at rung index 4. There is no measurement here that could have separated the two, and a free predictor that agrees with "the same position in the ladder we built" is not yet a predictor.

This is the Q6 move applied to a design rather than to an artefact. The 2.5% was computed from the two ladders before any endpoint was read, so the result was a test rather than a surprise.

What this actually establishes

A consistency check passed, and consistency checks are worth running. Had the two optima landed at different rung indices, the multiple-of-the-transition hypothesis would have been falsified for free. It was not.
It is not evidence for the predictor. Two ladders, one scaled by the candidate predictor itself, agreeing to 0.037 of a rung by construction.
The second candidate cannot be scored at all. R5 records the donor's final accuracy per rung (0.175 at 40 steps rising to 0.529 at 800) and P8 records none, so there is no second task to check it against. It is not refuted; it is untested.
The shared-donor optimum sits one rung later than the own-donor one, 400 against 240, which is R5's convergence finding showing up in a second place. P8 has no shared ladder, so this cannot be paired either.

The lesson, which generalises past this thread

Scaling an axis by a quantity to make two experiments comparable makes that quantity untestable as a predictor on those experiments. R5's most careful design decision is the one that removed R9's ability to answer its question from R5's data. Neither decision was wrong; they were made for different purposes a day apart, and nothing warned that the second depended on the first.

This is a close relative of what Q9 found, a unit that cancels out of its own ratio cannot answer a question about the axis it cancelled, and of Q8, where a grid tied two explanations together so tightly that any post-hoc story fitted every point.

What would settle it: R12

A third ladder at fixed absolute rungs, on a task whose control transition matches neither of these. Pick a task with a control transition near 120 and run P8's absolute ladder on it unscaled. Then 1.40x transition predicts about 168, landing on the 200 rung, while "rung index 4" predicts 120: a full rung apart, so the two hypotheses disagree and one of them loses. Cost is one donor ladder plus one receiver set, about the same as R5, and the receiver control needs no new machinery.

Limits

Two ladders, and one of them was built from the other. Everything above.
"The optimum" is the rung with the lowest break-even, not a continuous minimum. With seven rungs the resolution is coarse, and a true optimum anywhere between two rungs reads as whichever is nearer.
The own-donor and shared-donor columns disagree about where the optimum is, and this record uses the own-donor one throughout because that is P8's column. Anyone quoting a break-even for a shared donor should use the shared row and not this constant.
P8's break-evens come from its published JSON and were not recomputed here. They are the numbers its record cites and reproduce from that file exactly.
One width (48), one learning rate, six seeds, inherited from both parents.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
post hoc: Worked out after the fact, rather than decided in advance. We report such checks separately and never let them decide a result, because it is far too easy to find a pattern once you already know the answer.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.