How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Five Models Pay For One

By Joshua Bolick, Awareness Software Group ~13 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. Several of our recent results found that handing a trained model's internal machinery to a fresh one makes the fresh one learn much faster. Every one of those results carried the same warning: it is not a saving, because you have to train the donor first, and training the donor cost far more than it saved. This is the experiment that finally does the arithmetic.

What we found. It pays, but only if you are training several models and only if you keep the donor cheap. A donor has to get past its own turning point before it has anything worth giving; before that it passes on a handicap and actively slows the fresh model down. Just past that point is enough. Training the donor three times longer barely improves what it gives away while tripling what it costs, so the cheapest useful donor is the best one. At that setting a donor pays for itself once about five models have used it, and one donor shared between all of them works just as well as giving each its own.

Why it matters. The check worth copying here is the last one. It would have been easy to publish the break-even number without it: we had measured the benefit by giving every fresh model its own private donor, but the whole point of a break-even count is that one donor serves many. Those are different experiments, and only the second one supports the claim. So we trained a single donor and gave it to all six, and it worked exactly as well. Also worth stating plainly: this measures how much sooner a model starts performing, not how much less computing the whole job takes, and it is measured on models small enough to run on a laptop. It is a mechanism worth testing at a serious scale, not a technique anyone should deploy on our say-so.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 6 receivers across 7 donor lengths plus a shared-donor arm, no GPU, no cost.

Program v2 Bucket P, item P8. Decisive computation: analysis/donor_economics.py. Output: analysis/donor_economics.json. Reproduce with python analysis/donor_economics.py; --reuse re-derives every endpoint.

QUALIFIED 2026-09-01 by P9. The break-even number below holds at width 48 and nowhere above it. Swept across widths, the head start falls monotonically and its interval includes zero at width 96: -52.5 [-60.5, -44.5] at width 24, -27.5 [-32.1, -22.9] at 48 (reproducing this record), and -3.8 [-7.7, +0.2] at 96. As a share of the receiver's own run the same thing happens, -40.4% to -31.0% to -6.2%. So there is no saving to amortise at width 96, and the break-even of 4.8 receivers is undefined there rather than merely larger. The numbers below are unchanged and reproduce; what is withdrawn is any suggestion that they describe anything but a small model. The untrained-donor control still delays at every width, so the mechanism is intact and it is specifically the benefit of the donor having learned that disappears.

The qualification above is itself corrected, 2026-09-01, by P11. "Holds at width 48 and nowhere above it" was too strong. The head start only vanishes at width 96 when the task is left unchanged as the model grows; with difficulty matched to capacity it is -52.5 steps at width 96, larger in absolute terms than at width 48. The break-even count at that setting has not been recomputed -- the donor's cost rises with difficulty too -- so the honest statement is that the scope withdrawal was unwarranted and the economics at larger widths are open, not that they are known to hold.

The question

P1 through P7 established that a donor's converged recurrent machinery removes 35.8 steps from a receiver's time to transition, that the effect is irreducibly joint, and that six numbers recover about half of the bias third. Every one of those records says in its own Limits that it is not an efficiency claim, because the donor trained 400 steps to save 36.

**A donor has to finish learning first, and then stop.** A trained model can hand its internal machinery to a fresh one. Each bar is a different amount of training given to the donor before it did so. Green means the fresh model reached the point of suddenly performing sooner than it would have alone; red means later. The donor's own turning point falls at around 87 steps. A donor that has not finished learning passes on a handicap rather than a head start, and the damage is worst around forty steps. Just past its own turning point the benefit appears almost at once. After that, more training barely helps: going from 120 steps to 400 triples what the donor costs and buys only ten more steps of benefit, so the cheapest useful donor is also the most economical one. At 120 steps a single donor pays for itself once about five models have used it, and one donor shared between them works as well as giving each its own.

This is the record that prices it.

The originally filed framing was withdrawn before execution. P8 asked whether P7's six per-gate means could be had without a donor. That question cannot settle anything: P6 showed the bias contribution is interaction-only, so the six numbers do nothing without the donor's weights, and the donor's weights are the expensive part. The question that decides it is how short the donor can be, and how many receivers one donor can serve.

Kill test, fixed before execution: the cheapest donor producing any head start costs more than 10 receivers' worth of saving.

Anchors enforced in code, both passing: the 400-step donor reproduces P1's -35.8 [-42.0, -29.7], and a sub-transition donor still delays (+25.8).

What we found

A receiver left alone reaches its transition at about 86.7 steps.

Donor length	Receiver's head start	Receivers needed to pay for the donor
20	`+25.8` `[+20.7, +31.0]`	delays
40	`+29.2` `[+20.1, +38.2]`	delays
60	`+13.3` `[+1.1, +25.6]`	delays
80	`-5.0` `[-16.0, +6.0]`	interval spans zero
120	`-25.0` `[-30.7, -19.3]`	`4.8`
200	`-32.5` `[-38.0, -27.0]`	`6.2`
400	`-35.8` `[-42.0, -29.7]`	`11.2`

The kill test does not fire. A 120-step donor pays for itself across fewer than five receivers.

The donor has to get past its own transition, and barely past is enough

The sign flips between 60 and 120 steps, and the receiver's own transition sits at 86.7. A donor caught before it has learned anything transfers a handicap: +29.2 at 40 steps is a worse delay than at 20. Once past, the benefit arrives almost at once.

A longer donor is worse economics, not better

The saving saturates while the cost does not. Going from 120 to 400 steps triples the donor's price and buys 10.8 more steps, so the break-even count more than doubles, from 4.8 to 11.2. The cheapest useful donor is the best one, which is the opposite of what one would guess.

One donor serves everyone, which is the arm the economics actually rest on

Every row above gives each receiver its own donor. A break-even count of N assumes one donor serves all N, and those are different experiments. So one donor was trained at 120 steps, from a seed belonging to no receiver, and given to all six:

shared donor, 120 steps: -25.0 [-29.7, -20.3]

Indistinguishable from the bespoke curve's -25.0 [-30.7, -19.3]. A donor is not tuned to its receiver, and the amortisation is real rather than arithmetic about an experiment nobody ran.

What this is, and what it is not

It is the first efficiency claim in this programme with a break-even number attached. Under this programme's standing rule the instrument's cost is subtracted in the same units as the benefit, and here the donor is the instrument: 120 steps in, 25 steps out per receiver, positive from the fifth receiver onward.

It is not a claim about total training cost. The endpoint is time to the transition, not compute to a quality target. A receiver that transitions 25 steps early still has to finish training, and whether the whole run is shorter is a different measurement this record does not make.

And the scale is a toy. 17,280 parameters, one synthetic task, one width. What survives at scale is unknown, and the honest reading is that this is a mechanism worth testing bigger rather than a technique ready to use.

Limits

Time to transition, not total compute. Named above because it is the limit that matters most.
The shared arm tested one donor across six receivers, which is just above the break-even of 4.8. It shows the assumption holds at that size; it does not establish how far it extends.
The transition endpoint is quantised to 5 steps, so 80 steps reading -5.0 with an interval spanning zero is at the resolution floor and is reported as null rather than as a small effect.
All receivers share one task and one width. A donor for a different task keeps about half the benefit (P1), which is not priced here.
Six receivers, one donor seed per condition. A pilot.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
parameters: The adjustable numbers inside a model. Training is the process of setting them. Model size is usually quoted as a count of these.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.