How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

It Does Not Survive A Second Task

By Joshua Bolick, Awareness Software Group ~12 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Part of a bigger question: How do we know our own results are real? – Repeatedly, the control rather than the measurement decided the result -- and several striking findings dissolved when the right comparison was finally run.

In plain English

What we asked. There is a short window during a model's training where interrupting one part of it does far more damage than interrupting the same part earlier. Five of our results depend on that window existing, including the only genuine saving we have found. Every one of them was measured on the same exercise, and 'we only tried one exercise' is the most common caveat on this whole site. This is the experiment that tests it on a second one.

What we found. The window is clearly present on the familiar exercise and not detectably present on the second. The second exercise is a harder one where the model has to look up a different distance each time rather than always the same distance, and on that version the timing of the interruption stops mattering. So five of our results, including the saving, look specific to one exercise rather than general.

Why it matters. This is the counterweight to an otherwise good day. Another experiment had just shown that this same window survives being tested in much larger models, which had restored a conclusion we thought we had lost. Both were run to the same standard, and the honest summary is that the effect survives being made bigger and does not survive being moved. Worth mentioning how we nearly got this wrong. Our first attempt gave an answer so imprecise it could not tell 'no effect' apart from 'the same effect we found before', which we would have been within our rights to report as a negative and which would have meant nothing. The cause was that we measure the moment of learning as the halfway point between a run's own worst and best performance, and on the harder exercise the model was still improving when we stopped, so the halfway point kept moving. Using a fixed target taken from earlier measurements of that exercise made the answer three times more precise. The familiar exercise gave exactly the same number under both methods, which is what convinced us the fix was a fix rather than a convenience.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 2 tasks x 8 seeds x 3 arms, no GPU, no cost.

Program v2 Bucket L, item L7. Decisive computation: analysis/critical_period_task.py. Output: analysis/critical_period_task.json. Reproduce with python analysis/critical_period_task.py; --reuse re-derives every endpoint.

The question

D5, G2, J3, K3 and K5 are all measured on synthetic-delayed-copy-v1. "One synthetic task" is the single most common sentence in this programme's Limits sections.

**It does not survive a second task.** There is a short window during training where interrupting part of a model does far more damage than interrupting it earlier. Five of our results depend on that window existing, and all five were measured on the same exercise. This is the same measurement on that exercise and on a harder one, where the model must look up a different distance each time rather than always the same one. The window is clearly there on the familiar task and not detectably there on the second one. That is a real limit on five of our results, including the only genuine efficiency saving we have found, which now looks specific to one exercise rather than general. It is also the counterweight to a good day: a separate experiment had just shown that this same window survives being tested in much larger models. It survives being made bigger and does not survive being moved.

It matters more now than it did yesterday. P12's second case restored D5's scope across width, which puts K2, the only positive efficiency result here, back on D5's foundation. Task generality became the largest untested threat to it, which is why this was promoted above L6.

The second task is dispatch-copy-8lag, accepted by E1 because it has designed headroom: a plateau of 0.6175 against a chance floor of 0.0334. Its first token announces which of eight lags applies, so the model must dispatch rather than memorise one offset.

Kill test, fixed before execution: D5's paired endpoint is absent on dispatch-copy-8lag.

Anchor enforced in code: the delayed-copy arm must show a material placement effect with the same machinery. A null on a second task means nothing if the endpoint cannot see the effect where it is already established.

What we found

Endpoint is D5's: the delay from a freeze placed at the transition minus an identical-length freeze placed uniformly at random earlier in the same run. Materiality is K3's floor of 0.10 of the window.

Task	Transition	Final accuracy	Window	Placement effect	As a share of the window
`delayed-copy` (anchor)	`155.0`	`0.998`	`38.6`	`+10.4`	`+0.27` `[+0.20, +0.33]`
`dispatch-copy-8lag`	`296.6`	`0.576`	`74.1`	`+0.8`	`+0.01` `[-0.06, +0.08]`

The kill test fires, cleanly. The interval excludes the floor and excludes delayed-copy's value. There is no detectable critical period on the second task.

The first execution was inconclusive and is reported, not replaced

The first run used D5's own transition definition, a floor-to-ceiling midpoint from each run's own minimum and maximum, over 900 steps. It gave +0.10 [-0.10, +0.31] on the dispatch task: the kill test fired by its letter, and the interval spanned both zero and delayed-copy's +0.27, so it could not distinguish "absent" from "the same size and unmeasured".

The cause was checked rather than assumed. At K3's learning rate the dispatch task reads 0.482 at 900 steps, 0.588 at 2400, against E1's plateau of 0.6175. It is still climbing, so a midpoint between a run's own floor and its own ceiling chases a ceiling that keeps moving. That is M5's still-climbing confound and N5's warning about run-relative endpoints, in one place.

Two changes followed:

A fixed criterion. The transition is now the first step past halfway from that task's published chance floor to its published plateau, 0.5153 for delayed-copy, 0.3255 for dispatch: taken from E2's headroom module, which was built for this and had not been used this way. The level is a property of the task, not of the run.
A settled run length, 1800 steps rather than 900.

The interval narrowed from a width of 0.41 to 0.14, and the anchor is identical across both executions at +0.27 [+0.20, +0.33]. Delayed-copy is settled at either length, so the endpoint change moved the noisy arm and left the reference where it was. That is the check that makes the correction believable rather than convenient.

What this costs the programme

The critical period is a delayed-copy fact. Five causal records rest on it, and K2, the only positive efficiency result here, rests on D5. On this evidence that result is scoped to one task, and the scoping is real rather than a measurement artefact.

It is the counterweight to the day's restorations. P12 restored D5 across width; L7 limits it across tasks. Both were run against the same standard, and the honest position is that D5 survives being made bigger and does not survive being moved.

This does not say the phenomenon is absent everywhere. It says it is absent on the one alternative task tested, which was chosen because E1 accepted it and it has room to show harm. A second alternative could go either way, and "generalises" is not established by one task any more than "does not" is.

Limits

One alternative task. The strongest single statement available is that the effect is present on one task and absent on another, at the same width with the same machinery.
The dispatch task is genuinely different in kind, a dispatch on an announced offset rather than a fixed one, which makes it a strong test and also means a near neighbour of delayed-copy is untested.
Both arms at width 48, D5's and K3's width. Given P12, a width sweep on the second task is a natural follow-up and is not done.
The dispatch arm reaches 0.576 against a published plateau of 0.6175, so it is near but not at its ceiling even at 1800 steps.
Eight seeds per task. The dispatch interval is tight; the conclusion rests on it being tight.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
confound: A second explanation you did not control for. If bigger models both learn faster and score higher, then 'fast learners score higher' may be entirely about size and not about speed.
critical period: A stretch of time during which something has to happen for development to proceed normally. Borrowed from biology, where it describes windows in which a young brain must receive certain input.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
plateau: A stretch of training where the model is not visibly improving. Often, though not always, followed by a sudden jump.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.