How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

The Alarm Never Got To Speak

By Joshua Bolick, Awareness Software Group ~12 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Part of a bigger question: How do we know our own results are real? – Repeatedly, the control rather than the measurement decided the result -- and several striking findings dissolved when the right comparison was finally run.

In plain English

What we asked. One of our early-warning rules has to watch a model for 41 steps before it is allowed to raise an alarm, because it needs that much history to judge what is normal. We had tested it in progressively larger models and reported that it stops working in the largest one, and we drew a broad conclusion from that: this whole family of early-warning signals is an artefact of small models.

What we found. The conclusion did not follow. In that largest model the task was so easy that learning finished at step 39, which is two steps before the rule was allowed to say anything. It did not fail to warn us; it never got to speak. Give the same model a harder task, so there are nearly eighty steps of room after the warm-up, and both of our alarms work and give about fifty steps of warning. That is more warning than they give in a smaller model, not less.

Why it matters. This was the last of six results we set out to recheck, and it completes an uncomfortable picture: five of the six turned out to be problems with how we measured rather than facts about model size. The two instruments we had written off entirely both give more warning in bigger models once the task keeps pace with them, which is a coherent story from two unrelated measurements and the opposite of what our own archive said a day earlier. One caution we want to keep attached: an instrument that works is not the same as an instrument worth paying for. A separate result of ours found that acting on these warnings was matched exactly by acting at a random moment, and nothing here changes that.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 2 difficulty rungs x 5 seeds at width 192, no GPU, no cost.

Program v2 Bucket P, item P12, fourth and last case. Decisive computation: analysis/alarms_matched.py. Output: analysis/alarms_matched.json. Reproduce with python analysis/alarms_matched.py; --reuse re-derives every endpoint.

The question

K1 width-swept two gradient alarms and concluded that "the whole leading-indicator line is a small-model artifact". Its own verdict names the mechanism: at width 192 the model transitions before step 41, "which is the earliest this rule can fire at all", and its committed summary records room_before_warmup of -2.2 steps.

**The alarm never got to speak.** One of our early-warning rules has to watch a model for 41 steps before it is allowed to raise an alarm at all. We had reported it stops working in our largest model. The first pair of bars is that model on the original task; the second is the same model on a harder task. The red and green bars show how much training was left after the warm-up finished. A negative bar means the model had already learned the task before the rule was allowed to speak. On the original task there was no time left at all: the model finished two steps before the rule was permitted to fire, so of course it never warned us. Give the same model a harder task and there are nearly eighty steps of room, and the rule gives about fifty steps of warning, which is more than it gives in a smaller model. We had concluded from the first case that this whole family of early-warning signals was an artefact of small models. That conclusion did not follow from our own numbers, and the record now says so.

That is not an instrument failing. That is a run shorter than the instrument's warm-up. The alarm needs 41 steps of history before it can speak, and on the fixed lag-4 task a width-192 model has finished at step 38.8.

Kill test, fixed before execution: the alarms still fail to produce a usable lead at width 192 once the task is hardened.

Anchor enforced in code: the easy arm must reproduce K1's width-192 failure, both indicators unusable, room_before_warmup negative. It does, at -2.2.

K1's own guard is carried over unchanged, and it is the one that matters: a threshold firing at the same step in every run is a clock, not a detector, and would win a longest-lead contest by construction. choose_factor rejects thresholds whose firing step does not move.

What we found

Rung	Lag	Transition	Room before warm-up	Indicators with a usable lead
`easy` (K1's own)	4	`38.8`	`-2.2`	none
`hard`	11	`118.6`	`+77.6`	`global_norm`, `off_top`

Indicator, hard rung	Fires at	Lead
`global_norm`	`69.4`	`49.2` steps `[30.0, 68.4]`
`off_top`	`68.8`	`49.8` steps `[32.7, 66.9]`

The kill test does not fire. Both alarms work at width 192 once the run is long enough for them to fire at all, and both pass K1's clock guard, so the thresholds selected are genuinely moving with the run rather than firing on a fixed schedule.

And the leads are longer, not shorter. K1 measured 30.8 steps at width 48. At width 192 with difficulty matched, both indicators give about 49.

What this does to K1

K1's conclusion is withdrawn. "The whole leading-indicator line is a small-model artifact" does not follow from its own data: at the one width where the alarms failed, they failed because the model finished before they were allowed to speak. A correction banner is appended; K1's numbers reproduce exactly and the anchor here required that.

The narrower reading K1 could have drawn from the same numbers survives, and is worth keeping: an alarm with a fixed warm-up is unusable on a run shorter than its warm-up. That is a real constraint on deploying one, and it is a statement about the instrument's design rather than about the phenomenon.

P12 is complete, and the picture across all six

Result	What it reported	What the recheck found
P9 donor head start	closes with width	confound: flat when matched, `14x` larger at width 96
O7 weight spectrum	fades with width	withdrawn: a different flaw, the statistic moves with run length
K3 critical period	absent at width 192	confound, `0.27` of window against a `0.10` floor
J5 decode-probe lead	fraction unstable across widths	not the confound, but the lead grows, `14.8` to `39.9` steps
K1 two gradient alarms	leading indicators are a small-model artefact	confound: both recover, leads of `~49` steps

Five of six are measurement artefacts. One stands with its mechanism corrected.

The two instruments that were independently declared unusable at width, J5's decode probe and K1's gradient alarms: both give more warning in bigger models, not less, once the task keeps pace. That is a coherent picture from two unrelated measurements and it is the opposite of what the archive said this morning.

Limits

Five seeds per rung, one width, one task family. The leads carry wide intervals ([30.0, 68.4]) and should be read as "the alarm works" rather than as a calibrated number.
Width 192 is still a small model. What is restored is that these instruments survive capacity, not that they survive scale.
Difficulty is raised through lag only, as in every case of P12.
The hard rung transitions at 118.6 against a target near 130; the match is approximate.
This does not revive the economics. J7 found acting on a warning was matched exactly by acting at a random time, and nothing here touches that. An instrument that works is not the same as an instrument worth paying for.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

confound: A second explanation you did not control for. If bigger models both learn faster and score higher, then 'fast learners score higher' may be entirely about size and not about speed.
critical period: A stretch of time during which something has to happen for development to proceed normally. Borrowed from biology, where it describes windows in which a young brain must receive certain input.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
probe: A small separate model trained to read information out of a bigger model's internals, used as a measuring instrument rather than as a product.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

The plain-language version. This page is the record: the design, the numbers and the limits. If you would rather have the story, The Alarm Never Got To Speak, explained without the jargon, covers the same result for a reader who has never seen a training curve.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.