How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 45 of 51

The Window Closes as Models Grow

By Joshua Bolick, Awareness Software Group ~15 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. These models do not improve gradually. They sit at chance for a while and then get good quite suddenly, and we had found that just before that moment there is a short window during which one part of the model is doing work that cannot be done at any other time. Hold that part still during the window and the model takes noticeably longer to learn; hold it still for the same length of time at some random earlier moment and it barely matters. That difference is the window. We had only ever measured it at one model size.

What we found. It shrinks as models get bigger, and at the largest size we tested it is gone entirely. At that size, holding the part still at the supposedly critical moment is no worse than doing it at random. The middle size reproduces our original number exactly, so this is a limit on where the finding applies rather than a contradiction of it. We also checked the obvious objection, that we measured over a shorter stretch in bigger models: expressed as a fraction of each model's own timeline the effect is steady across the small sizes and only then collapses, which is not what a measurement shrinking along with its own ruler would look like.

Why it matters. The part worth your time is how nearly we published the opposite. Our first three attempts at this all reported that the window was absent at every size, including the size where we had originally found it. Each looked like a clean result. What caught them was a single check we ran alongside: repeat the original experiment exactly and confirm it still gives the original answer. It did not, three times running, and each time for a different reason. The last and real one was that our original code held the part still by one method and the new code used another, which sounds equivalent and is not. The written description of a method says what was done; only the code says which of several things that means. When someone shows you a replication, the useful question is whether they re-ran the original alongside it.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 144 training runs, no GPU, no cost.

Program v2 Bucket K, item K3. Decisive computation: analysis/critical_period_width.py. Output: analysis/critical_period_width.json. Reproduce with python analysis/critical_period_width.py, or re-derive the endpoints from the saved series with --reuse.

The question

J5 width-swept the predictive side of this programme and the result was bad: the decode probe's crossing collapses from step 301.6 to step 5.5 across widths 16 to 192, until at the widest it fires at step 5 of a 34-step run and carries almost no timing information.

**The moment when timing matters disappears as models get bigger.** These models have a short window, just before they suddenly get good, during which one part of them is doing work that cannot be done at any other time. We measured that by holding the part still during the window, and separately holding it still for the same length of time at a random earlier moment, and taking the difference. A tall bar means the timing mattered. A bar at zero means holding it still at the critical moment was no worse than doing it at random, so there was no special window at all. The window shrinks steadily as the model grows and has closed completely by the largest size tested, where the bar sits at zero. The effect was already known at the middle size and this reproduces it there exactly, so this is a limit on where that finding applies rather than a contradiction of it. It also matches what we found separately about our early-warning signal, which degrades with size in the same direction: both of the structures this project is built on point the wrong way for bigger models.

**The effect is not just shrinking along with how we measured it.** Bigger models here learn sooner, so the window we measured was scaled to each model's own timeline rather than fixed. That raises an obvious objection: perhaps the effect only looked smaller in bigger models because we measured over a shorter stretch. This chart answers it by expressing the same result as a fraction of each run's own length. If the objection were right, this line would be flat. It is flat, and then it is not. Across the three smaller sizes the effect is a steady six to seven percent of the run, which is what a measurement scaling with its own ruler looks like. Then it falls and goes below zero. The collapse at the large end is therefore a real change in the models rather than an artefact of how we measured, which is the whole reason for reporting both versions of the number.

Nobody had width-swept the causal side. D5's critical period, G2's 61%-in-one-direction and J3's 6.5x off-diagonal asymmetry are all width-48 facts. And K2, the programme's only positive efficiency result, rests on D5.

Kill test, fixed in advance: D5's paired endpoint, the cost of a targeted freeze minus the cost of a random-time-matched one, does not vary with width beyond seed noise.

Result: the critical period shrinks with width and is gone by 192

Eight seeds per width, D5's configuration, window proportional to each run's own transition.

Width	Transition	Targeted	Random	Paired (targeted − random)	As share of transition
16	`272.8`	`+28.6`	`+11.2`	`+17.4` `[+7.9, +26.9]`	`+6.5%`
32	`209.1`	`+25.2`	`+11.2`	`+14.0` `[+11.9, +16.1]`	`+6.8%`
48	`155.2`	`+19.0`	`+7.8`	`+11.2` `[+8.9, +13.6]`	`+7.3%`
96	`90.0`	`+9.0`	`+4.1`	`+4.9` `[+3.6, +6.2]`	`+5.4%`
192	`50.5`	`+2.5`	`+4.0`	`-1.5` `[-4.0, +1.0]`	`-3.0%`

The kill test fires. The paired effect varies 18.9 steps across the ladder against 3.6 steps of seed noise, a ratio of 5.3x, and at width 192 it is not material at all: freezing the recurrence at the transition is indistinguishable from freezing it at a random earlier moment, and if anything marginally better.

Width 48 reproduces D5 exactly (+11.2 against D5's +10.6), so this is scope rather than disagreement, the same relationship J5 has to I2.

It is not the shrinking window

The window is proportional to each run's transition, so it shrinks with width (68 steps at width 16, 13 at width 192). The obvious alternative reading is that the effect shrank because the window did.

The share column rules that out. If the effect merely tracked window length, its share of the transition would be constant. Instead it is flat across the narrow end, 6.5%, 6.8%, 7.3%, and then collapses to 5.4% and -3.0%. A flat-then-collapsing share is the signature of a real change in the phenomenon, not of a measurement scaling with its own instrument.

A fixed window cannot be used instead: at width 192 the transition is at step 50, so two non-overlapping 40-step windows do not fit before it and the wide end of the ladder is structurally unmeasurable that way.

Three failed attempts, and the anchor that caught all of them

This record exists because the harness was wrong three times, and each time it produced a confident, publishable-looking answer. D5's own fixed-window design at width 48 was run as an anchor, and it returned +6.2, then +4.2, then +2.9, against D5's +10.6 [+8.3, +13.0], every interval spanning zero. Each run reported "no material critical period at any width", including width 48, where D5 found one.

The causes, in the order they were wrongly diagnosed:

Logging resolution. Every measured delay was a multiple of 5, the evaluation interval, and the effects were 0-5 steps. Real, and fixed by evaluating every step. Not the cause.
Learning rate. J5's 0.005 was used so the ladder would match J5's; D5 ran at 0.002, where the width-48 transition is at 155 rather than 85. Real, and fixed. Still not the cause.
The freeze itself. D5 zeroes the gradient inside the window. The reimplementation cleared requires_grad. Under AdamW these are different interventions: zeroing leaves the optimiser stepping every parameter so the arms differ only in what the step contains, while clearing requires_grad makes it skip the parameter and suspends its momentum and decay as well. D5's code carries a comment saying exactly this. D5 also clips gradients, which had been omitted.

With the freeze corrected the anchor returns +11.0 [+8.4, +13.6] against D5's +10.6 [+8.3, +13.0], and the sweep above becomes interpretable.

The generalisable lesson: a record's prose says "freeze the component". There are several defensible ways to do that and they differ here by a factor of three. To re-test a result, read its code rather than its record, and make the anchor a precondition in code, because the first version printed the sweep and the failing anchor side by side, and the sweep read as a finding.

Verdict

The kill test fires. The critical period's size is a fact about model size, and at width 192 it is absent altogether.
D5 is scoped, not overturned. Width 48 reproduces it. Every result resting on D5's placement sensitivity inherits the scope, which includes G2.
The causal side scales the same way the predictive side did. J5 found the probe's lead collapsing with width; this finds the critical period collapsing with width. Both of the programme's central structures point the wrong way for scaling up, on the same ladder, measured independently.
K2 is partly insulated. Its saving rests on the recurrence continuing to contribute accuracy long after the transition, which is a different claim from placement sensitivity, and it was shown to need no transition knowledge at all. But it too was measured only at width 48 and is untested elsewhere.

Limits

Eight seeds, five widths, one task, one learning rate, one component. Only the recurrence was frozen; D5 also mapped the input, embedding and head, and those are not swept here.
Width and transition time are confounded by design. A fixed learning rate means wider models transition sooner (273 steps at width 16, 50 at width 192), which is J5's confound inherited deliberately: screening a rate per width would confound size with whatever the screen optimised for. The share column is what separates "the effect scales with the run" from "the effect scales with width", and it is flat before it collapses.
Width 192's transition at step 50 is close to the floor where a proportional window and its random counterpart can both be placed. The absence there is measured, but it is measured at the edge of what this design can do.
The window is proportional, D5's was fixed. The anchor establishes the two agree at width 48; it does not establish they would agree everywhere.
This says nothing about why. A critical period that shrinks with width is consistent with F3's reservoir effect, a wider untrained recurrence is already a better feature extractor, so less of the mechanism has to be built during training, but nothing here tests that.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
AdamW: A widely used training algorithm that adapts how big a step it takes for each individual parameter. The default in most modern AI training.
confound: A second explanation you did not control for. If bigger models both learn faster and score higher, then 'fast learners score higher' may be entirely about size and not about speed.
critical period: A stretch of time during which something has to happen for development to proceed normally. Borrowed from biology, where it describes windows in which a young brain must receive certain input.
freezing: Stopping one part of a model from changing while the rest carries on learning. A way of asking what that part contributes, by taking it away.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
probe: A small separate model trained to read information out of a bigger model's internals, used as a measuring instrument rather than as a product.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.