How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

A Head Start From Another Model

By Joshua Bolick, Awareness Software Group ~14 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Part of a bigger question: Which parts of a model actually matter? – The parts that move most are not the parts that matter, and no single component is required -- the model routes around every freeze. What a part is worth shows up only when you remove it.

In plain English

What we asked. Models here spend a long time doing badly and then improve sharply. We wanted to know whether the thing a finished model has built inside itself can simply be handed to a fresh one, so the fresh one does not have to build it. An earlier experiment of ours tried this with a single training step and found it was worth almost nothing in another model. This time we handed over the finished internal machinery instead.

What we found. It works, and it is a large effect: the fresh model reaches the point of suddenly performing about forty per cent sooner. It is genuinely about the learning rather than about being given somebody's numbers, because machinery from a model that had not finished learning, or had never learned at all, made things worse rather than better. But two checks say the gift is not personal. Scrambling which internal unit is which makes no difference whatsoever, and machinery trained on a different task still gives half the benefit.

Why it matters. The useful part of this is what it rules out. Because scrambling the units changes nothing, whatever is being handed over cannot be a specific piece of wiring. It has to be some overall property of a trained model's numbers that survives being shuffled, and we do not yet know which property. That is a cheap next experiment and it is already written down. It is also worth being clear that this is not a saving: we trained a donor for four hundred steps to save a receiver thirty-six. It would only become one if a single donor could help many models, or if the useful property could be manufactured without training a donor at all. We publish it because it corrects one of our own earlier conclusions in an interesting direction, not because it makes anything cheaper today.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 6 receivers with their own donors across 7 arms, no GPU, no cost.

Program v2 Bucket P, item P1. Decisive computation: analysis/machinery_transplant.py. Output: analysis/machinery_transplant.json. Reproduce with python analysis/machinery_transplant.py in about twenty minutes on a throttled laptop CPU; --reuse re-derives every endpoint from the saved series.

CORRECTED 2026-09-01 by P5. The kill test below was badly chosen, and the conclusion drawn from it is too strong. This record reads the identical permuted arm as "nothing donor-specific moved". That does not follow. A relabelling of hidden units, applied consistently, preserves the donor's computation exactly -- it renames units, and the receiver's readout is still training, so it adapts to the new names within a few steps. The permuted arm was therefore never capable of destroying donor-specific content, and its agreement with the converged arm is close to arithmetic rather than evidence. P5 settles it from the other side. Imposing the donor's coordinate-free statistics on a random initialisation -- spectrum, gate-norm distribution, gate biases, overall gain, and all of them together -- reproduces none of the head start; every arm mildly delays, and all gives +4.2 [-4.9, +13.2] against this record's -35.8. So the transferable content is not a summary statistic. The corrected reading: the donor's converged recurrent structure transfers, and it is the structure that matters, not its labelling and not its statistics. The numbers below are unchanged and reproduce; only the interpretation of the permuted arm is withdrawn. This is a case of the trap named in CLAUDE.md: a rule's grid containing its own components as degenerate points.

The question

J8 is this programme's cleanest negative on transfer: the leading gradient direction is causally load-bearing inside its own run and delivers 3-7% of that effect in another, indistinguishable from a random donor. A 2026 preprint (arXiv 2606.25010) reports the opposite for a different object: patching converged attention structure from a post-emergence checkpoint into a pre-emergence model reportedly elicits the capability.

**A head start from another model, and what it is not.** We took the internal machinery of a model that had already finished learning and dropped it into a fresh model that had not. Green bars mean the fresh model got to the point of suddenly performing sooner than it would have alone; red bars mean it took longer. Each variation changes one thing about where the donated machinery came from. Finished machinery gives a real head start, about forty per cent of the time it normally takes. Handing over machinery that has not finished learning, or has not learned at all, makes things worse rather than better, so this is not simply the benefit of being given somebody else's numbers. But scrambling which internal unit is which changes nothing at all, and machinery trained on a different task still delivers half the benefit. Whatever is being handed over is not specific to the donor, and only about half of it is specific to the task.

Those are not in contradiction. A gradient direction is one step; converged attention is a structure. J8 may have failed because it transferred the wrong kind of thing.

Kill test, fixed before execution: a receiver given a donor's converged recurrent machinery reaches its transition no earlier than an untouched control, or no earlier than the same weights with their hidden units relabelled.

Anchor, enforced in code. J8's transfer fraction re-derives from its committed series to 3.4% and 6.7%, inside the published 3-7%. The script exits 2 without interpreting anything if it does not.

What we found

Steps to the transition against each receiver's own control. Negative means sooner.

Arm	What it copies	Effect
`recurrent`	the donor's converged recurrent block	`-35.8` `[-42.0, -29.7]`
`recurrent-permuted`	the same weights, hidden units relabelled	`-35.8` `[-41.0, -30.7]`
`recurrent-wrong-task`	a donor converged on a different lag	`-19.2` `[-21.3, -17.0]`
`recurrent-and-readout`	recurrent block plus the readout	`-30.0` `[-46.6, -13.4]`
`everything`	the whole donor	`-56.7` `[-61.0, -52.4]`
`recurrent-pre-transition`	the donor before its own transition	`+28.3` `[+22.0, +34.7]`
`recurrent-untrained`	a third initialisation, never trained	`+24.2` `[+19.0, +29.3]`

Held-out accuracy ends at 0.983-0.989 everywhere except the pre-transition arm (0.967), so none of this is bought by damaging the model.

Something real travels

-35.8 steps is large: the control transitions at about 86, so a donor's converged recurrent block removes roughly forty per cent of the time to get there. And it is not "being handed foreign weights": an untrained donor of the same shape delays the transition by +24.2, and a donor caught before its own transition delays it by +28.3. Only converged weights help.

This is the thing J8 did not find. A gradient step from another run was worth 3-7% of its own effect. A converged structure from another run is worth thirty-six steps.

It is not the donor

The relabelled arm is identical: -35.8 [-41.0, -30.7] against -35.8 [-42.0, -29.7]. Permuting which hidden unit is which changes nothing at all. So the kill test fires, on its second half, and it fires for the right reason: whatever transfers is invariant to the donor's choice of coordinates, so nothing donor-specific has moved. That is J8's conclusion arriving again by a different route.

About half of it is about this task

A donor converged on a different lag still delivers -19.2 [-21.3, -17.0], roughly half the effect, with intervals well separated from the same-task donor's. So the transferable content splits into a generic part, converged recurrent weights are simply a better starting point than random ones, whatever they learned, and a task-specific part of about the same size.

What this leaves

A permutation-invariant property of converged recurrent weights carries about forty per cent of the time-to-transition, and we do not know which property. Spectrum, weight-norm distribution, gate-bias statistics and effective gain are all invariant to relabelling and all candidates. That is a specific, cheap next experiment and is filed as P5.

It also matters that this is not an efficiency claim. The donor was trained for 400 steps to save the receiver 36; end to end that is worse than not doing it. It becomes interesting only if one donor accelerates many receivers, or if the carried property can be synthesised without training a donor at all, which is exactly what P5 would tell us.

The control that had to be fixed, and why the first version said nothing

The first execution reported recurrent and recurrent-permuted as identical to the decimal, which prompted a check rather than a write-up. Two things were wrong:

The permutation also permuted the columns of the input-to-hidden matrix. Those are embedding dimensions, not hidden units. That is not a relabelling of units; it scrambles which input feature each gate reads. Corrected to permute columns only for the hidden-to-hidden matrix.
More importantly, the arm was not testing what it claimed. The receiver keeps its own embedding and readout, so a donor's recurrent block sits between foreign surroundings in every arm. There is no donor-to-receiver unit correspondence for a permutation to destroy.

The corrected permutation still gives an identical result, so the conclusion is unchanged, but it is now supported by a control that tests the thing it names. The two arms that carry the argument, recurrent-untrained and recurrent-wrong-task, were added at the same time and are what make the positive result interpretable at all.

Limits

One task family, one architecture, one width. The wrong-task donor differs only in its lag; a genuinely different task would be a stronger separation.
The donor is transplanted at step 30, one fixed point. Effects may depend on where in the receiver's run the graft lands, and that is not swept.
Optimizer state is not transplanted, deliberately: it is the donor's bookkeeping rather than its machinery, and J8 is the record of what happens when bookkeeping is moved. A run that transplanted it too would be a different experiment.
everything is an upper bound and a sanity check, not a result: copying the whole donor is the donor.
Six receivers. The intervals are paired within seed, which helps, but this is a pilot.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
checkpoint: A saved copy of a model partway through training, so you can come back and inspect what it looked like at that moment.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
held-out: Data the model was never trained on, kept back specifically to test it. Scoring a model on data it has already seen measures memorisation, not learning.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
optimizer: The algorithm that decides how to change the model given the gradient. AdamW and SGD are two common choices and they behave differently.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.