How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 49 of 67

The Check That Could Have Sunk It

By Joshua Bolick, Awareness Software Group ~14 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. Our longest-standing useful measurement is one that spots a model about to improve well before it actually does: we change one earlier word at a time and see which position the prediction leans on. It finds the right position very early. But a separate result of ours raised an awkward possibility. The exercise we measured it on is one where the answer is always the same distance back, so there was only ever one right position to find, and the measurement might have been telling us about the exercise rather than about the model.

What we found. So we moved the target. In a harder exercise the first word of each sequence announces how far back to look, and it differs every time. We lined the measurement up with each sequence's own announced distance and ran it again. The early warning did not just survive, it got longer -- the model settles on the right position about a hundred and fifty rounds before its answers start improving -- and it got cleaner, with almost all of the model's dependence sitting exactly on the answer instead of about half of it.

Why it matters. The check that decides this is the one we ran deliberately wrong. We repeated the whole measurement misaligned by one position, and the entire signal moved by one position with it, never once resting on the true answer. That is what tells us the measurement is tracking each sequence's own answer rather than some fixed habit of the model. Two other things worth saying. This also explains an oddity from the easier exercise, where a fully trained model seemed to lean heavily on words that could not help it -- that turns out to be a quirk of that exercise, not of models like ours. And our first attempt at this failed its own consistency check, for a reason worth knowing if you ever compare training runs: we measure the moment of improvement as the halfway point between a run's worst and best scores, and a run stopped earlier has a lower best score, so its halfway point arrives earlier. Nothing was wrong; the two runs were simply different lengths. We matched the lengths and it agreed.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 6 training runs (12 including the discarded first execution), no GPU, no cost.

Program v2 Bucket N, item N5. Decisive computation: analysis/dispatch_dependence.py. Output: analysis/dispatch_dependence.json. Reproduce with python analysis/dispatch_dependence.py, or re-derive every endpoint with --reuse.

The question

M3 found the longest lead in this programme: a GRU's largest dependence sits on a valid copy source +83.3 steps before its accuracy midpoint, from a ranking rather than a threshold.

**Where the prediction's dependence sits, once the answer moves around.** On this harder task the answer is not always the same distance back. The first word of each sequence announces how far to look, and it differs from sequence to sequence. So the measurement is lined up with each sequence's own announced distance, and position zero means 'exactly where this example's answer is'. The blue bars are that measurement. The orange bars repeat it deliberately misaligned by one place, as a check on whether the lining-up is doing any work. Almost all the dependence sits exactly on the answer's position and nowhere else -- much cleaner than on the easier task, where about half of it sat on words that could not help. The control is the important half of the picture: misaligned by one place, the whole peak moves by one place, which is what should happen and confirms the measurement is tracking each sequence's own answer rather than some fixed habit. The model finds the right place to look very early, long before it can use it.

M1 then found the task it was measured on is solved outright by a zero-parameter cache model. On synthetic-delayed-copy-v1 the answer is always exactly three places back, so "which position matters most" has one fixed answer for every example, and a probe that finds it early may be reading a property of the task rather than of the model.

dispatch-copy-8lag removes that. The first token announces the lag, drawn per example from seven possibilities, so the answer's position changes from sequence to sequence and the stream is uniform random rather than periodic. That also means there is exactly one valid source, where delayed-copy's periodicity gave two.

Kill test, fixed in advance: no lead survives when the source position is not constant across examples.

Design: the probe has to be re-keyed

M3 corrupts a token a fixed number of places before the scored position. Here each example's answer sits at its own announced distance, so the corruption goes at t − lag(example) + d for a relative offset d, making d = 0 the true source in every sequence regardless of its lag. A fixed-offset probe would smear seven different alignments together, and that is what the kill test is really about.

Three controls, all in code before the run:

the untrained model, evaluated before the first update;
a mis-keyed probe, aligned to lag + 1 instead of lag, if the alignment is doing the work, mis-keying it must move the signal;
a threshold-free endpoint beside the thresholded one, plus a sweep of the dependence floor.

The anchor failed first, and the reason is the finding's own foundation

The first execution ran 800 steps and produced a midpoint of 136.7 [128.1, 145.2] against M2's 174.0 [162.9, 185.1]. No overlap. Nothing had failed to reproduce.

**The midpoint is where accuracy crosses halfway between the run's own floor and ceiling, and on this task accuracy is still climbing at step 800, 0.449 at 400, 0.499 at 600, 0.537 at 800, against a 0.6175 plateau at 2,000. A shorter run has a lower ceiling, a lower halfway level, and therefore an earlier** midpoint.

STEPS was set to M2's 1,200, taken from M2's stated configuration rather than searched over to make the anchor pass. The midpoint then landed at 158.3 [146.1, 170.6], which overlaps M2's interval, and the diagnosis is confirmed rather than assumed: every per-seed midpoint rose with the longer run (140, 170, 150, … against 130, 140, 130, …).

A floor-to-ceiling midpoint is not comparable across runs of different lengths. That is now a standing rule, and it applies to any endpoint normalised by a run's own total change, sharpness included.

Result: the kill test does not fire

Mean dependence by relative offset, six seeds, 1,200 steps. 0 is each example's own announced source.

stage	−3	−2	−1	0	+1	+2	+3
untrained	`+0.002`	`−0.002`	`−0.001`	`+0.001`	`+0.002`	`+0.003`	`+0.002`
step 100	`−0.004`	`−0.006`	`−0.004`	`+0.181`	`+0.007`	`+0.000`	`+0.000`
step 400	`+0.006`	`+0.003`	`+0.014`	`+0.365`	`+0.011`	`+0.004`	`+0.009`
final	`−0.007`	`−0.020`	`+0.001`	`+0.465`	`−0.004`	`−0.006`	`−0.007`
final, mis-keyed	`−0.010`	`−0.006`	`−0.005`	`−0.007`	`+0.465`	`−0.007`	`−0.016`

endpoint	value
accuracy midpoint	`158.3` `[146.1, 170.6]`
correct source ranked first from	`10` in all six runs
lead of that ranking	`+148.3` `[+136.1, +160.6]`
same, mis-keyed control	never: it does not rank the true source first at any step
dependence first clears `0.05`	`13.3` `[7.9, 18.8]`
lead of that threshold	`+145.0` `[+132.1, +157.9]`
final share on the true source	`0.960` `[0.918, 1.002]`

The lead survives, and it is longer than on the easy task, +148.3 against M3's +83.3, and against F2's +23.5 at 55.4% of the training budget.

Every control behaves. Untrained models rank the correct source first in 0/6 runs, against a 1-in-7 chance, with total dependence 0.0153. The mis-keyed probe moves the entire peak by exactly one position, +0.465 at d = +1, −0.007 at d = 0, so the alignment is doing the work and the probe is tracking each sequence's own answer rather than a fixed habit. The floor sweep runs 151.7, 145.0, 128.3, 86.7 across a tenfold change and never approaches zero.

And the profile is much cleaner than delayed-copy's. 0.960 of measurable dependence sits on the true source here against M3's 0.479. Delayed-copy's periodicity gave a second valid source and its repeating structure gave the model reason to track tokens that could not help; neither is present here. The useless-token dependence M3 found is a property of that task, not of the architecture, which is worth noting beside N1, where blocking a gradient pathway did not reduce it.

Verdict

The kill test does not fire. The ranking leads by +148.3 steps [+136.1, +160.6] on a task whose source position varies per example.
The anchor reproduced once the run length was matched, and the failure that preceded it is documented rather than removed.
The mis-keyed control is decisive: the peak moves with the keying, and the mis-keyed arm never ranks the true source first at any step of any run.
The untrained control is flat at 0/6 against 1-in-7.
Dependence is far more concentrated on this task (0.960 against 0.479), so M3's useless-token finding is about delayed-copy, not about GRUs.

Limits

The detection step is run-length invariant; the lead is not. The correct source is ranked first from step 10 in every run at both run lengths, but the lead is that step subtracted from the midpoint, and the midpoint moves with run length (+126.7 at 800 steps, +148.3 at 1,200). Quote the detection step alongside the lead, or quote the lead with its run length attached.
The anchor overlaps but the means differ (158.3 against M2's 174.0), on different seeds. Interval overlap is the stated test and it passes; a tighter comparison would need shared seeds.
Six seeds, one task, one width, one learning rate, one lag set.
Nothing here says the lead is worth anything. K1 showed the manoeuvre a lead is spent on works just as well timed at random, and K2 showed the one real saving needs no detector. A longer lead on a harder task does not change either; it establishes that the instrument measures the model rather than the task.
The probe costs seven forward passes per reading (fourteen with the control), which is not free. It is not priced against a benefit here because no benefit is claimed.
d = 0 being the only valid source is a property of this task's uniform random stream. A task with any repeating structure would reintroduce M3's second source.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
plateau: A stretch of training where the model is not visibly improving. Often, though not always, followed by a sudden jump.
probe: A small separate model trained to read information out of a bigger model's internals, used as a measuring instrument rather than as a product.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.