How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 17 of 39

The Forty Steps That Matter

By Joshua Bolick, Awareness Software Group ~13 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. A model that learns suddenly has to be building something beforehand. We wanted to know which part of it, and when. So we stopped one part at a time from learning for a short stretch, then let it carry on, and measured how much later the model got there.

What we found. One part out of four has a window in which it has to be working: the model's memory. Pause it in the forty steps leading up to the moment of learning and the model takes roughly twice as long to get there compared with pausing it earlier. Pause the same part forty steps *after* that moment and the cost is essentially zero. The other three parts do not care when you pause them at all, and one of them, the model's output layer, can be switched off entirely for that stretch without changing anything.

Why it matters. This is a genuine critical period, of the kind biologists describe in development, and it is narrow: the part goes from being the most timing-sensitive thing in the model to irrelevant within forty steps. It also matters that nothing was permanently damaged. Every version still learned the task to the same accuracy in the end. We changed when learning happened, not whether it could.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, no GPU, no cost.

Program v2 tier 4, item D5. Decisive computation: analysis/freeze_window.py. Output: analysis/freeze_window.json. Reproduce with python analysis/freeze_window.py in about forty minutes on a throttled laptop CPU.

The question

D1 froze components for whole runs and found none is required: the model routes around every freeze, and the finding lives in what each one costs. D5 asks a different question, is there a critical period?

**Only one part of the model cares when you hold it still.** We stopped one part of the model from learning for 40 steps and then let it resume, and measured how much later the model then learned the task. Each part was tested twice: with the pause placed immediately before the moment of learning (pink), and with the same length of pause placed at a random earlier point (grey). The whiskers are 95% confidence intervals across eight runs. For three of the four parts the two bars match, so pausing them costs the same wherever you do it. The recurrent part, the model's memory, is the exception: pausing it just before the model learns costs more than twice as much as pausing it earlier. That is what a critical period looks like. Note also the rightmost pair, the output layer, where both bars are exactly zero: you can switch off the model's readout for 40 steps and the timing of learning does not notice.

**The damage peaks right at the moment of learning.** The previous figure showed that pausing the model's memory just before it learns costs more than pausing it earlier. But 'just before' is also simply 'later', so this figure separates the two. The same 40-step pause is slid along the training run, from 160 steps before the moment of learning to 40 steps after it. Zero on the horizontal axis means the pause ended exactly when the model learned. The cost rises to a maximum near zero and falls away after it. That is a genuine critical period: there is a window in which the memory has to be learning, and interrupting it then is worse than interrupting it at any other time, including later. If the cost had simply kept climbing, the earlier result would have been about training progress rather than about the moment of learning.

Freeze one component for a 40-step window just before the transition, then release it. If some component has to be learning during that window, freezing it there should cost more than freezing it anywhere else.

Kill test, from docs/research-program-v2.md: every freeze delays equally, or none does. Then no component is a bottleneck and the transition is a genuinely distributed event.

Why this is answerable now

Placing a window "just before the transition" requires knowing when the transition will be. F1's phase diagram made that predictable from settings, but this pilot does better: training is deterministic given a seed, so each seed's own control run supplies its own transition, and the frozen run shares that trajectory exactly up to the step the freeze starts. All 8 seeds transitioned, between steps 141 and 163.

The control that decides it

D3 established that any perturbation delays the transition by about 21 steps, almost regardless of its size. So a freeze will delay the transition, and a study reporting that would be reporting D3.

Every targeted window is therefore matched with a random-time-matched window of identical length, placed uniformly at random earlier in the same run. The endpoint is the paired difference between the two, not the delay.

Evaluation runs every step rather than on the usual 4-step grid, because the endpoint is a difference between two delays and the grid would quantise a large fraction of it.

Part A: only the recurrence cares where it is frozen

Component	Targeted delay	Random-placed delay	Paired difference	Material
recurrent	`+19.1` `[+18.2, +20.1]`	`+8.5` `[+6.5, +10.5]`	`+10.6` `[+8.3, +13.0]`	yes
input	`+10.8` `[+9.6, +11.9]`	`+10.5` `[+8.0, +13.0]`	`+0.2` `[-2.5, +3.0]`	no
embedding	`+0.9` `[+0.2, +1.6]`	`+1.1` `[+0.2, +2.1]`	`-0.2` `[-0.8, +0.3]`	no
head	`+0.0` `[+0.0, +0.0]`	`+0.0` `[+0.0, +0.0]`	`+0.0`	no

The kill test does not fire. Freezing the recurrent matrices just before the transition delays it 2.2x more than freezing them at a random earlier time, and the paired interval sits well clear of zero. Nothing else shows any placement sensitivity at all.

Two internal checks come out right without being aimed at:

The head is completely inert. Freezing the output layer for 40 steps changes the transition time by exactly zero, at either placement, in all 8 seeds. That matches D1's finding that the readout is not load-bearing, arrived at by a different route.
The input matrices are placement-blind. They delay substantially (+10.8) but identically wherever the window sits, so a large delay does not by itself imply a critical period. The input arm is what stops "recurrent delays a lot" being mistaken for the finding.

Final accuracy is unaffected in every arm (0.986–0.988, against controls at 0.987). This is a timing result, not a damage result: every arm still learns the task, 8/8.

Part B: a peak, or a ramp?

Part A's control has a structural limitation worth stating plainly: **the random window is always earlier than the targeted one.** A component whose freeze cost simply grows as training proceeds would produce part A's result exactly, with nothing special about the transition at all.

So the same freeze is swept across fixed offsets relative to each seed's own transition, from 160 steps before it to 40 steps after.

A critical period is a peak near zero offset, with cost falling again afterwards.
A ramp is monotone in offset, cost still rising at the last measured placement.

It is a peak, and the fall on the far side is the sharpest thing in this record.

Freeze window ends at	Delay	8 seeds
transition `-80`	`+6.4` `[+5.1, +7.6]`	8/8
transition `-40`	`+12.9` `[+11.9, +13.8]`	8/8
the transition	`+19.1` `[+18.2, +20.1]`	8/8
transition `+40`	`+0.1` `[-0.2, +0.4]`	8/8

Cost climbs steadily as the window approaches the transition, triples between -80 and 0, and then collapses to nothing. Forty steps after the transition, freezing the recurrent matrices for forty steps delays learning by a tenth of a step, with an interval containing zero.

So the recurrence goes from the most placement-sensitive component in the model to completely irrelevant inside forty steps. That is not a cost that grows with training progress, which was the alternative part A could not exclude; a ramp would still be rising at the last placement, and this one has fallen to zero.

The two offsets furthest back (-160, -120) could not be measured: with transitions at steps 141 to 163, those windows begin before training does. They are recorded as skipped rather than as zero.

What this joins up with

Four independent lines now say the same thing about the recurrence, none of them designed to confirm the others:

E3: the phenomenon is gated recurrence, absent in attention.
F3: the transition builds the mechanism rather than selecting it, a frozen random recurrence never gets there.
D1: the recurrence is load-bearing and the readout is not.
D5: and the recurrence must be learning during a specific window, which closes at the transition.

D5's head arm independently reproduces D1's result by a different route, +0.0 delay, both placements, all 8 seeds, which is some evidence these are four views of one fact rather than four restatements of one artifact.

Limits

One architecture, one task, one width, one learning rate, one batch size, eight seeds.
The window length (40 steps) is a single choice, and it is about a quarter of the typical transition time here. A much shorter or much longer window is a different experiment.
The random-placement control draws from [10, transition - 80], which is a wide range covering most of pre-transition training; it establishes that placement matters, and part B rather than part A is what characterises how.
Freezing is implemented by zeroing gradients, so the optimizer still takes a step for every parameter and the arms differ only in what that step contains. Weight decay is off throughout, for the reason D1 recorded: AdamW's decoupled decay shrinks a parameter whether or not it has a gradient, so a frozen group under decay would drift toward zero and the arm would measure decay.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
AdamW: A widely used training algorithm that adapts how big a step it takes for each individual parameter. The default in most modern AI training.
architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
batch size: How many examples the model looks at before updating itself once. Bigger batches give a steadier but more expensive update.
critical period: A stretch of time during which something has to happen for development to proceed normally. Borrowed from biology, where it describes windows in which a young brain must receive certain input.
freezing: Stopping one part of a model from changing while the rest carries on learning. A way of asking what that part contributes, by taking it away.
gated: A design where the model has explicit internal switches deciding what to remember and what to discard at each step. GRUs and LSTMs are gated; plain recurrent networks are not.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
optimizer: The algorithm that decides how to change the model given the gradient. AdamW and SGD are two common choices and they behave differently.
phase diagram: A map showing what a system does across combinations of settings, so you can see where behaviour changes rather than testing one point at a time.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
weight decay: A common training setting that gently pulls a model's internal numbers toward zero, used to stop it over-fitting.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

The plain-language version. This page is the record: the design, the numbers and the limits. If you would rather have the story, The Forty Steps That Matter, explained without the jargon, covers the same result for a reader who has never seen a training curve.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

Why we do this. The same care shown above is what we apply to client work. We build custom AI models that you own and run yourself, using our own platform, and the reason we publish research like this is that a model is only worth what its evaluation is worth. If a supplier cannot show you the experiments that went against them, you cannot check the ones that went their way.

All research from this programme →