How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 46 of 51

The Warning Runs Out of Room

By Joshua Bolick, Awareness Software Group ~12 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. We have two ways of getting an early warning that one of these models is about to learn its task. One reads the model's internal state. The other watches how much its internal adjustments jitter from step to step. We already knew the first one gets worse in bigger models, so this asks about the second.

What we found. It gets worse too, and then it stops working altogether. In small models it gives a warning about eighty steps ahead, which shrinks to about thirty as the model grows. Beyond that we could not get a usable warning out of it at all, and the reason is mechanical rather than mysterious. The signal needs a stretch of history to compare against before it can say anything, and that takes a fixed number of steps no matter how big the model is. Bigger models learn sooner. In the largest one we tested, the model has already learned before the detector is able to speak at all.

Why it matters. The two signals fail in opposite ways, which is the useful part. The first becomes useless by firing almost immediately, so its warning covers nearly the whole run and tells you nothing. The second cannot fire early enough. One runs out of room at the end, the other at the start, and both stop being useful at exactly the sizes anyone would care about. Together with a separate result about the internal structure they were built to anticipate, which also fades with size, everything we have built here works best on the small models it was developed on. That is worth saying plainly rather than discovering later, and it is the sort of thing a research programme usually finds out only when somebody else tries to use it.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 25 training runs, no GPU, no cost.

Program v2 Bucket K, item K4. Decisive computation: analysis/gradient_indicator_width.py. Output: analysis/gradient_indicator_width.json. Reproduce with python analysis/gradient_indicator_width.py, or re-derive the endpoints with --reuse.

The question

J5 width-swept F2's decode probe and found it degrades badly with size. There are two indicators and only one had been tested. A11's global_norm:variance and G1's off_top:variance had never been width-swept, and the question mattered both ways: if they hold their lead the programme should switch instruments; if they collapse too, the entire leading-indicator line is a small-model artifact.

**The warning needs a fixed run-up, and the event keeps arriving sooner.** Our second early-warning signal watches how much the model's internal updates jitter from step to step. To say anything it first needs a stretch of history to compare against, which takes a fixed number of steps no matter how big the model is. The flat orange line is that run-up. The blue line is when the model actually learns, which happens sooner in bigger models. The two lines cross. In the largest model tested, the thing the detector exists to predict has already happened before the detector is able to say anything at all. This is the mirror image of what we found for our other early-warning signal, which fails in big models by firing almost immediately and so telling you nothing. One runs out ofroom at the start, the other at the end, and both stop being useful at exactly the sizes anyone would care about.

Kill test, fixed in advance: the gradient indicator's lead varies with width beyond seed noise.

Result: the kill test fires, and then stops being measurable at all

Five seeds per width, K1's alarm rule, threshold swept per width.

Width	Transition	Crossing	Lead (steps)	Lead (fraction)
16	`177.8`	`93.0`	`+84.8`	`0.474`
32	`110.2`	`61.0`	`+49.2`	`0.445`
48	`88.4`	`57.6`	`+30.8`	`0.348`
96	`56.2`	,	no usable threshold	,
192	`38.8`	,	no usable threshold	,

Both statistics behave identically to within a step. Across the three widths where a lead can be measured at all, it varies 54 steps against 10.8 of seed noise, a ratio of 5.0x for global_norm and 5.3x for off_top. The kill test fires.

The anchor passes. K1 measured this rule at this exact configuration and published width-48 leads of +31.2 and +29.8; this pilot returns +30.8 [+26.4, +35.2] and +30.4 [+26.8, +34.0]. The instrument is the one K1 used.

Why the wide end is blank, which is the actual finding

"No usable threshold" is not a shrug. The rule cannot fire before step 41: a 20-step rolling window must fill, and the running median it thresholds against needs a history longer than that window. That number is a property of the detector and is fixed in steps. The transition it is trying to anticipate moves earlier as width grows.

Width	Transition	Room the detector has
16	`177.8`	`+136.8`
32	`110.2`	`+69.2`
48	`88.4`	`+47.4`
96	`56.2`	`+15.2`
192	`38.8`	`-2.2`

At width 192 the model has already learned before the detector is able to speak. At width 96 there are fifteen steps of room, which is not enough for any threshold to fire in every run and still vary between them.

The two instruments fail in opposite ways

This is the part worth carrying:

J5's probe fails by firing too early. F3's reservoir effect means a wider untrained recurrence is already a better feature extractor, so the answer becomes linearly decodable almost immediately: at width 192 the probe crosses at step 5.5 of a 34-step run and its "warning" covers nearly the whole remaining run.
The gradient alarm fails by not firing early enough. Its warm-up is a fixed number of steps, and the event it watches for arrives sooner and sooner until it arrives first.

Opposite mechanisms, same destination: an instrument that cannot anticipate anything. The leading-indicator line is a small-model artifact, and J5's negative generalises beyond the probe.

Verdict

The kill test fires on both gradient statistics, at 5.0x and 5.3x seed noise.
And it understates the problem, because the two widest points are not a degraded lead but no measurable lead at all.
Every instrument this programme has built degrades with width. J5 for the probe, K4 for both gradient statistics, and K3 for the causal structure they were meant to anticipate. Three independent measurements on the same ladder, all pointing the same way.
This does not say early warning is impossible at scale. It says these three instruments, at these settings, stop working, and it identifies a specific, fixable reason for one of them.

Limits

Five seeds per width, one task, one learning rate, one alarm family. The rule is K1's median-ratio threshold, not A11's or G1's.
A11 and G1 used different instruments, at rates 0.001-0.003, a 30-step window and an IQR threshold calibrated to a 20% surrogate false-alarm rate. Their published leads (+18.9, +35.4) are not the numbers this reproduces and were deliberately not used as the anchor, because anchoring on them would fail for reasons unrelated to width. K1 is the anchor because K1's configuration is this one.
The warm-up limit is a property of the window length, and the window was not swept. A shorter window would fire sooner and the wide end might become measurable. Whether the lead would survive that is untested, and it is the obvious follow-up: a detector whose warm-up scales with the run rather than being fixed in steps. Nothing here shows that would work, only that the fixed-window version cannot.
Width and transition time are confounded by design, as in J5 and K3: a fixed learning rate means wider models transition sooner. The fraction column is reported for the same reason J5 reports it, and it falls too (0.474, 0.445, 0.348), so the collapse is not only the denominator moving.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

false-alarm rate: How often a warning system goes off when nothing is actually happening. Any detector can be made to fire early by making it twitchier, so a lead time only means something alongside this number.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
probe: A small separate model trained to read information out of a bigger model's internals, used as a measuring instrument rather than as a product.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
surrogate: A fake version of real data, built to keep everything except the effect being tested. If a detector fires just as often on the surrogate, it is responding to something other than what it claims.
variance: How spread out a set of numbers is. Rising variance in a signal means it is becoming more erratic.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.