How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

One Forward Pass Tells You

By Joshua Bolick, Awareness Software Group ~17 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: Which parts of a model actually matter? – The parts that move most are not the parts that matter, and no single component is required -- the model routes around every freeze. What a part is worth shows up only when you remove it.

In plain English

What we asked. Our previous result found that copying part of a finished model into a fresh one can roughly halve how long the fresh model takes to learn, or nearly double it, depending entirely on whether the two jobs were related. We flagged that rather than claiming it, because we had only three examples and we had decided what counted as 'related' after seeing the answers. That is exactly how people fool themselves, and we had done it once already a few pages earlier.

What we found. So this time we wrote down the test first. Before running anything, we picked one cheap check: take the finished model, run it once on the new job with no training at all, and see how well it does. Then we ran every combination of four different jobs, sixteen pairings in all, and asked whether that one number predicted what happened.

Why it matters. It did, and more cleanly than we expected. Every pairing where borrowing helped had the finished model already doing better than chance on the new job. Every pairing where borrowing cost time had it doing no better than chance. There is a clear gap between the two groups with nothing sitting in it. So there is a usable rule here: if the model you were thinking of borrowing from cannot beat chance on your task, borrowing from it will leave you worse off than starting from nothing, and one quick run tells you before you commit. One nice detail is that the relationship has a direction. A model trained on the harder version of a job helps with the easier one much more than the other way round, which a simple notion of similarity would get backwards. We also say plainly which row of our results we do not trust and show that the conclusion does not depend on it.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, a 4x4 donor-by-receiver matrix over 6 seeds, no GPU, no cost.

Program v2 Bucket Q, item Q14. Decisive computation: analysis/relatedness_predicts_transfer.py. Output: analysis/relatedness_predicts_transfer.json. Reproduce with python analysis/relatedness_predicts_transfer.py in about twenty-five minutes on a throttled laptop CPU; --reuse re-derives every endpoint from the saved series.

The question

Q4 found a donor's converged recurrent block worth -53.2% of a receiver's time to the transition on the same task, -20.7% from a related task, and +85.7%, a large delay, from an unrelated one. Three donors, ordered by how related the tasks looked.

**Borrowing from a model trained on the wrong task is worse than borrowing nothing.** Sixteen combinations. We take a model that has finished learning one task, copy part of its internals into a fresh model, and time how long that fresh model then takes to learn a task. Down the diagonal the two tasks are the same. Everywhere else they differ. Negative numbers mean the fresh model got there faster than one given nothing at all; positive means slower. The last column is the whole result. A model that had thoroughly learned counting makes every copying task slower, by up to 89% of the time the fresh model would have needed on its own. Before running any of this we checked each finished model once on the new task, which costs a single run. That one number separated every helpful pairing from every harmful one with nothing in between, so you can tell in advance whether borrowing will pay. The bottom row is the exception we do not read: that task is learned so quickly that the copy arrives too late to matter, and all four of its cells come out the same.

Q4 flagged that rather than claiming it, because "shares a shape" was applied after the tasks were chosen and any monotone story fits three points. Q8 is the record of what happens when a post-hoc pattern that fits every point in a small grid is preferred over a preregistered test; Q11 withdrew it the same day.

So this is the version that can fail.

The measure, fixed before execution. relatedness(D -> R) is the converged donor D's zero-shot held-out accuracy on task R, as the fraction of the way from R's chance floor to R's plateau. One forward pass, no training, and available before any decision to graft, which is what would make it a decision rule rather than a description.

Kill test, fixed before execution. Either clause fires it:

Ordering: across the twelve cross-task cells, the rank correlation between relatedness and the transfer effect is not negative.
Separation, the relatedness of the cells that help overlaps the range of those that harm.

Prior recorded in advance: ordering passes, separation is the clause at risk, because zero-shot accuracy is coarse and the distant cells should cluster at chance. The prior was wrong about which clause was at risk. Separation is the stronger of the two.

Result: the kill test does not fire on either clause

Steps to the transition against each receiver's own control. Negative means sooner. Diagonal cells are same-task and are excluded from the test.

Receiver (control)	`delayed-copy` donor	`dispatch` donor	`noisy-copy` donor	`running-count` donor
`delayed-copy` (`86.7`)	`-36.7`	`-12.5`	`-34.2`	`+55.0`
`dispatch-copy-8lag` (`122.5`)	`-20.8`	`-55.0`	`-14.2`	`+109.2`
`noisy-copy` (`91.7`)	`-40.0`	`-17.5`	`-39.2`	`+57.5`
`running-count` (`37.5`)	`+20.8`	`+20.8`	`+20.8`	`+20.0`

And the measure, computed before any of it:

Donor \ Receiver	`delayed-copy`	`dispatch`	`noisy-copy`	`running-count`
`delayed-copy`	`0.999`	`0.176`	`1.014`	`-0.227`
`dispatch-copy-8lag`	`0.499`	`1.031`	`0.506`	`-0.230`
`noisy-copy`	`0.992`	`0.186`	`1.007`	`-0.229`
`running-count`	`-0.055`	`-0.090`	`-0.060`	`1.000`

Ordering: rho = -0.754 over the twelve cross-task cells, the predicted sign.

Separation: perfect, with room to spare. Every cell that helps has relatedness between 0.176 and 1.014. Every cell that harms has relatedness between -0.230 and -0.055. The gap is 0.232 and nothing sits in it.

The decision rule this licenses

If a candidate donor scores at or below chance on your task, it will cost you more than starting from scratch. One forward pass tells you, before you spend anything.

That is a usable rule rather than a description, which is the whole point of fixing the measure in advance. Its magnitude is not small: at the extreme, a running-count donor costs a dispatch receiver +109.2 steps against a control that reaches the level in 122.5.

The row I do not trust, and what happens without it

running-count as a receiver is not interpretable and I am saying so before using it. Its control reaches the level at 37.5 steps and the graft lands at step 30, so every arm in that row is timed from a point after most of its learning. The four arms read +20.8, +20.8, +20.8 and +20.0: within 0.8 steps of each other. That is what "the graft costs the same whatever it carries" looks like, not a relatedness effect, and it means the row measures disruption.

Three of the twelve cross-task cells come from it. So:

	Cells	`rho`	Separates	Gap
all cross-task cells	`12`	`-0.754`	yes	`0.232`
dropping the `running-count` receiver row	`9`	`-0.883`	yes	`0.232`

The verdict does not need the suspect row. Removing it makes the correlation stronger and leaves the separation gap identical, because the cells doing the separating are the ones where a running-count donor is given to a copy task, and those are in the other three rows.

Two things the measure does that a similarity intuition would not

It is not symmetric, and the asymmetry is right. A dispatch donor scores 0.499 on delayed-copy while a delayed-copy donor scores only 0.176 on dispatch. Dispatch is the harder task and a model that solves it has, in effect, also solved the fixed-offset case; the reverse does not hold. The transfer effects follow: -20.8 for dispatch-into-delayed-copy against -12.5 for delayed-copy-into-dispatch. Relatedness between tasks is a direction, not a distance, and a symmetric intuition would have got this pair backwards.

It exceeds 1.0 where the receiver's own ceiling is capped. A delayed-copy donor scores 1.014 on noisy-copy, above the noisy-copy control's plateau, because that plateau is held down by 15% label noise while the donor has the un-noised structure exactly right. The measure is not bounded above and does not need to be.

The endpoint, and the sign that survives changing it

The level is one constant per receiver task, pooled over that task's control runs: halfway from the first reading to the plateau the controls reach. It is a task-level number applied identically to every arm, so it is not N5's per-run trap.

Read instead under L7's published task criterion, for the three tasks that have one, the magnitudes move a long way and every sign and every ordering is unchanged:

Receiver	`delayed-copy`	`dispatch`	`noisy-copy`	`running-count`
`delayed-copy`	`-36.7`	`-15.0`	`-34.2`	`+51.7`
`dispatch-copy-8lag`	`-40.8`	`-90.0`	`-33.3`	`+145.0`
`noisy-copy`	`-39.2`	`-17.5`	`-38.3`	`+57.5`

Magnitudes are a property of the level; the ordering is not. Given how much of Bucket Q was about definitions deciding answers (Q2, Q6, Q7), a result whose conclusion is definition-independent is worth marking as such.

The anchor fired twice, and both were mine

Q4's same-task dispatch arm reproduces at -90.0, exactly. It took two corrections to get there and neither was a problem with Q4.

First, I read Q4's number at this pilot's level and got -55.0. That is precisely the error Q4's own record warns about in as many words: the effect widens with the level it is read at, so any number for it is a number at a level. Reproducing a result means reproducing it under its own endpoint, not under the one you happen to be using. The anchor caught it; without the anchor in code I would have concluded Q4 did not replicate and gone looking for a cause that did not exist.

Second, the cross-task cell read -40.8 against Q4's -35.0. Also not a failure. Q4 trained its delayed-copy donor for 400 steps, because that donor doubled as its own anchor donor, and this pilot holds every donor at 800 steps, because a relatedness comparison across four tasks cannot also vary the donor budget by task. A more converged donor helps more. That cell is now reported rather than gated on, with the reason in the code.

What stands

A measure fixed in advance separates helpful donors from harmful ones perfectly, across twelve cross-task cells with a gap of 0.232 and nothing in it, at a cost of one forward pass.
It orders them too, rho = -0.754, rising to -0.883 once the row that measures graft disruption rather than relatedness is dropped.
Q4's flagged pattern survives being made falsifiable, on four task families rather than three and with the measure written down first.
Relatedness is directional. The measure gets an asymmetric pair right that a symmetry intuition gets backwards.

Limits

Four task families, one width (48), one learning rate, one graft step (30). The graft step is what makes the running-count receiver row uninterpretable, and a version timed as a fraction of each receiver's own control would fix it.
Twelve cross-task cells is twelve points. The separation is clean and the gap is wide, but this is a small grid, and a small grid is exactly what Q8 was about. What makes this different from Q8 is that the measure was fixed before execution and the kill test could have fired.
All four tasks share a vocabulary and a sequence length, deliberately, so that a donor can be evaluated on another's data without reshaping. Whether the measure survives a change of shape is untested.
Zero-shot accuracy is one candidate measure, not the only one, and its success here does not establish that it is the best. It was chosen for being free.
Not an efficiency claim. Every donor trains for 800 steps to save a receiver at most 90.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
gated: A design where the model has explicit internal switches deciding what to remember and what to discard at each step. GRUs and LSTMs are gated; plain recurrent networks are not.
held-out: Data the model was never trained on, kept back specifically to test it. Scoring a model on data it has already seen measures memorisation, not learning.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
plateau: A stretch of training where the model is not visibly improving. Often, though not always, followed by a sudden jump.
post hoc: Worked out after the fact, rather than decided in advance. We report such checks separately and never let them decide a result, because it is far too easy to find a pattern once you already know the answer.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
vocabulary: The set of distinct symbols a model can read and produce.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.