How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Growth Did Not Travel

By Joshua Bolick, Awareness Software Group ~10 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. Our last result was a real saving: start a model small and grow it during training, and it reaches the target with a quarter less compute. A result on one task is not yet a result, so we tried the identical recipe on a second, slower task.

What we found. It mostly failed: the growing model rarely reached the target at all, and when it did it cost more than a model trained at one fixed size. The likely reason is that the recipe grew the model at moments chosen for the fast task, long before this slower task had been learned. We have marked the earlier result as one-task only and are testing timings scaled to the slower task.

Why it matters. The lesson: a recipe tuned on one task carries that task's timescale with it. Rescale it before trusting it elsewhere.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 27 schedule-tuning runs of 3000 steps; the measured phase did not run because the precondition failed. The design and kill test were committed (e22c485) before any run, with a throwaway result disclosed in advance.

Program v2 Bucket O, item O27. Decisive computation: analysis/progressive_width_sum.py (O5's code with the task and budget changed). Output: analysis/progressive_width_sum.json.

The question

O5 found that growing a GRU 24 -> 48 -> 96 reaches held-out accuracy 0.95 with 26% fewer FLOPs than the cheapest fixed width, on a copy task. O27 reran it unchanged on a second, slower task (modular sum, E5's task), with 3000 steps, the schedule re-tuned over O5's nine (s1, s2) pairs on three throwaway seeds. Disclosed before running: on one throwaway seed width 48 reached 0.95 at step 1660 for 52 G FLOPs (width 96: step 1120, 133 G), and one growth schedule reached only 0.49.

**On a second, slower task, the growing model rarely reached the target at all.** The growth recipe that saved a quarter of the compute on the first task, tried on a task that takes about seven times longer to learn. Each bar is one choice of when to grow; its height is how many of three trial runs reached 95% accuracy. Only the latest growth times ever reached the target, on one run each, and those runs cost more than simply training a fixed-size model. The growth timings were chosen for the faster task; the next test scales them to this one.

Result: the precondition fails; the kill test is not interpreted

Growth schedule `(s1, s2)`	Throwaway seeds reaching `0.95`	FLOPs where reached
`(50, 100)`, `(50, 150)`, `(50, 250)`, `(100, 150)`, `(100, 200)`	`0/3`	--
`(100, 300)`	`1/3`	`258` G
`(200, 250)`	`1/3`	`325` G
`(200, 300)`	`1/3`	`195` G
`(200, 400)`	`1/3`	`167` G

No schedule reached the target on every throwaway seed, so none could be chosen and the receivers were not run. Across all 27 runs, 4 reached 0.95, all on the same seed, each at more compute than either fixed width used in the throwaway check (52 and 133 G).

Why, as far as this record can say

The growth grid was built for the other task's timescale. O5's copy task is learned in about 150 steps, and its grid grows the model between steps 50 and 400. This task takes about 1000 steps at width 48, so every schedule grows the model long before it has learned anything, and the latest schedules do best. The grid was carried over unscaled -- a design that decided part of its answer, noticed on reading the result.
The tuning runs saved only their FLOPs to target, not their curves, so what the failing runs did (learn slowly, stall, or never start) cannot be read back. This breaks the standing rule to save raw series first; the fix belongs in O28.

What stands

Precondition fails; the kill test is not interpreted.
O5's saving does not carry over as designed: on the second task, growing on O5's schedules reached the target in 4 of 27 runs, each more expensive than a fixed width. O5 carries a banner saying so.
Generates O28: the same comparison with the growth grid scaled to this task's timescale and every run's curve saved.

Limits

Three throwaway seeds, one task, one growth rule. Whether growth fails here because of its timing, the optimiser restart, or the task is not separated.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
held-out: Data the model was never trained on, kept back specifically to test it. Scoring a model on data it has already seen measures memorisation, not learning.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.