How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 26 of 39

The Layer That Changes Most Is Not the One That Matters

By Joshua Bolick, Awareness Software Group ~10 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. Models are often built by stacking layers. We had already found that one part of a single-layer model has a narrow window, about forty steps before it learns, in which it must be working. This asks what happens when there are two layers: does each have its own window, and do they sit at the same moment?

What we found. They differ, but not in the way we expected. The two layers do not have windows at different times. One layer has a window and the other simply does not. Pausing the layer nearer the input matters much more if you do it just before the moment of learning than if you do it earlier. Pausing the layer nearer the output costs about twice as much overall and makes no difference at all when you do it. Both stop mattering entirely once the model has learned.

Why it matters. The interesting part is the mismatch with an earlier result. We had found that the layer nearer the output is the one whose internals reorganise most during learning. It turns out to be the layer without the window. So the part that changes most is not the part with the critical timing, which is the fourth time in this programme that two reasonable-sounding ways of finding the important part of a model have pointed at different parts. This page also records a mistake: our first reading of the data said something different, and it came from treating the largest number in a noisy set as meaningful when it was not distinguishable from its neighbours.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, no GPU, no cost.

Program v2 Bucket G, item G3. Decisive computation: analysis/depth_critical_period.py. Output: analysis/depth_critical_period.json. Reproduce with python analysis/depth_critical_period.py in about fifty-five minutes on a throttled laptop CPU.

The question

D5 found a critical period: freezing the recurrence for the forty steps before the transition costs +19.1 steps, against +8.5 at a random earlier time, collapsing to +0.1 forty steps after. That was measured on a single-layer model.

**Only one layer cares when you pause it.** A model with two stacked layers of memory. Each was stopped from learning for forty steps, twice: once immediately before the moment the model learned the task (pink) and once at a randomly chosen earlier point (grey). If a layer has a critical window, its pink bar should stand clearly above its grey one. Whiskers are 95% confidence intervals across six runs. The two pairs behave completely differently. For the layer nearer the input, pausing just before the moment of learning costs clearly more than pausing earlier, so it has a real window. For the layer nearer the output, the two bars are the same height: pausing it hurts about twice as much overall, and it does not care at all when you do it. So the layers differ in kind rather than in timing. The twist is that a separate experiment found the layer nearer the output is the one that reorganises most during learning, and it turns out to be the one without the window. What changes most is again not what matters most.

E6 then found the expansion concentrates near the readout, nearly doubling from layer 0 to layer 1. So a single-layer model is the deep end of a deep one, and D5 measured that end without anyone choosing to.

Does each layer have its own critical period?

Kill test: both layers show the same window, peaking at the same offset with the same magnitude. Then depth adds no temporal structure to the critical period either.

Result: they differ in kind

Depth 2, learning rate screened on excluded seeds, 6 of 6 seeds usable.

Layer	Frozen just before	Frozen at a random earlier time	Paired difference	Material
0 (near input)	`+3.5` `[+2.4, +4.6]`	`+0.5` `[-1.5, +2.5]`	`+3.0` `[+0.4, +5.6]`	yes
1 (near readout)	`+7.5` `[+4.8, +10.2]`	`+6.8` `[+5.0, +8.6]`	`+0.7` `[-0.9, +2.2]`	no

Only layer 0 has a critical period. Layer 1 costs roughly twice as much to freeze in absolute terms and does not care when you do it, freezing it at a random earlier moment costs +6.8 against +7.5 targeted.

The offset sweeps say the same thing from the other side:

Window ends at	Layer 0	Layer 1
transition `-80`	`+0.0` `[-3.9, +3.9]`	`+7.2` `[+5.2, +9.2]`
transition `-40`	`+4.2` `[+2.1, +6.2]`	`+6.5` `[+4.8, +8.2]`
the transition	`+3.5` `[+2.4, +4.6]`	`+7.5` `[+4.8, +10.2]`
transition `+40`	`-0.2` `[-0.6, +0.3]`	`-0.2` `[-0.6, +0.3]`

Layer 0 rises from nothing at -80. Layer 1 is flat across the whole pre-transition span. Both collapse to -0.2 after the transition, so both matter before it, and only one cares when.

That is D5's own recurrent-versus-input pattern reappearing one level up: D5 found its input matrices delayed substantially but identically wherever placed, which is exactly layer 1's behaviour here.

The correction this record exists to record

The first verdict said the opposite, and it was decided by argmax. It reported "different windows: the layers peak at different offsets", because layer 0's largest value sits at -40 and layer 1's at 0. Checking whether those peaks separate from their own neighbours:

Layer 0's peak +4.2 [+2.1, +6.2] against its value at 0, +3.5 [+2.4, +4.6], overlapping.
Layer 1's three pre-transition values, +7.2 / +6.5 / +7.5, flat.

Neither peak is distinguishable from its neighbour. That was argmax on noise, which is precisely the error E8 was written to avoid three items earlier, made again here.

The verdict now decides on D5's own paired endpoint, targeted against random placement, and the output JSON records peaks_separate_from_neighbours: false so the argmax reading cannot be resurrected from the data.

The inversion

E6 found the expansion concentrates in layer 1. G3 finds the critical period is in layer 0.

The layer that reorganises most is not the layer with the window. That is D1's prominence-against-necessity dissociation appearing in a third place, after G1 found the causal gradient direction is not the predictive one.

The programme has now separated the same cluster of properties four times, on four different objects, and they have never once agreed.

Verdict

The kill test does not fire, but not for the reason it anticipated. The layers do not differ in when their window sits; they differ in whether they have one.
Layer 0 has a genuine critical period (+3.0 [+0.4, +5.6]); layer 1 is placement-blind (+0.7 [-0.9, +2.2]) while costing about twice as much to freeze at any placement.
Both collapse after the transition, so the window closes for both.
D5 was measured on the placement-blind end. Its +19.1 is a single-layer number and the single-layer model is not simply "layer 1 of a deep one", the deep-end layer here shows the weaker placement effect, not the stronger.

Limits

One task, one width, one optimizer, six seeds, depth 2 only. Depth 3 is untested and E6 found it seed-fragile at the rates that make depth 2 work.
The absolute delays here are much smaller than D5's (+3.5 and +7.5 against +19.1), because freezing one layer of two leaves the other free to compensate. The comparison that matters is between the layers, both measured identically, not against D5's number.
The learning rate had to be screened for room rather than speed: the transition must be late enough to fit a targeted window and a random earlier one. The first attempt used E6's speed-first rule, chose a rate transitioning at step 78 against a floor of 110, and found 0 of 6 seeds usable. That is the guard working, and it is why the screen now requires both.
Four offsets is a coarse sweep. A finer one might resolve a peak in layer 0 that this design cannot, though it would not change layer 1's flatness.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

critical period: A stretch of time during which something has to happen for development to proceed normally. Borrowed from biology, where it describes windows in which a young brain must receive certain input.
freezing: Stopping one part of a model from changing while the rest carries on learning. A way of asking what that part contributes, by taking it away.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
optimizer: The algorithm that decides how to change the model given the gradient. AdamW and SGD are two common choices and they behave differently.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

Why we do this. The same care shown above is what we apply to client work. We build custom AI models that you own and run yourself, using our own platform, and the reason we publish research like this is that a model is only worth what its evaluation is worth. If a supplier cannot show you the experiments that went against them, you cannot check the ones that went their way.

All research from this programme →