How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 35 of 39

The Check We Should Have Run First

By Joshua Bolick, Awareness Software Group ~10 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. This project has published thirty-three write-ups about something that happens inside a model at the moment it learns a task. All thirty-three measured models that were learning. Not one of them measured a model that trained for exactly as long and learned nothing at all. That is a real gap, because without it we could not tell our claim apart from a much more boring one: that this simply happens whenever you train a model of this kind, learning or not.

What we found. So we trained a second set of models on exactly the same inputs, with the same settings and the same starting points, but with the correct answers replaced by random noise. There was nothing to learn, and they learned nothing: they finished no better than guessing. The effect we have been reporting all along did not appear in them. The models that learned showed it at seventeen to twenty-five times the size, and the ones that could not learn stayed well inside the range we had agreed in advance was too small to count.

Why it matters. We stacked the deck against ourselves on purpose. The models that could not learn have no particular moment to examine, so we let each one report its single best moment anywhere in training, which is the most flattering measurement possible and is guaranteed to look larger than it should. They still showed nothing. We expected this outcome, and we think it was worth the compute anyway: it was the cheapest experiment available that could have shown our central claim to be about something else entirely, and a project that has never run that experiment is not entitled to say so.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 16 training runs plus screening, no GPU, no cost.

Program v2 Bucket I, item I6. Decisive computation: analysis/unlearnable_control.py. Output: analysis/unlearnable_control.json. Reproduce with python analysis/unlearnable_control.py, or re-decide the endpoint from the saved series with --reuse.

The question, and why it should have been asked first

Every expansion measurement in this programme has been taken on a model that learns something. Not one had been taken on a model that trains for the same number of steps, on the same inputs, with the same optimiser, and learns nothing at all.

**The internal signature does not appear when there is nothing to learn.** Everything this project has measured about a moment of learning was measured on models that were learning. That leaves an obvious hole: does the internal change we keep reporting actually need learning, or does it happen whenever you train one of these models at all? So we trained a second set on the exact same inputs with the answers replaced by random noise, so there was nothing to learn. The shaded band is the size we agreed in advance was too small to count. Nothing happens without learning. The two grey bars are models that trained just as long and ended no better than guessing, and they sit deep inside the too-small band, seventeen to twenty-five times below the models that learned. We deliberately gave the grey bars the most flattering possible measurement, letting each one report its single best moment anywhere in training, and they still show nothing. We expected this result, and it was worth running anyway: it was the cheapest experiment that could have shown our main claim to be about something else entirely.

Until that exists, "the expansion accompanies acquisition" has no evidence separating it from "the expansion accompanies training a GRU on anything". Thirty-three records rest on a distinction nothing in the record had tested.

The unlearnable arm keeps everything except the learnability: same vocabulary, same sequence length, same batch size, same optimiser, same schedule, same seeds, same inputs. Only the targets are replaced with fresh uniform noise at every step, so there is no function to fit.

Kill test: the unlearnable arm shows a material expansion somewhere in its trajectory. Then the signature is a fact about training a GRU rather than about acquisition, and a large part of this record changes meaning.

The arms behave as the design requires

Arm	Final held-out accuracy	Transitioned
learnable	`0.9909`	8/8
unlearnable	`0.0317` (chance is `0.0312`)	0/8

The unlearnable arm sits at chance after 700 steps. It learned nothing, which is the point.

Result: the control holds, by a factor of 25

The unlearnable arm has no transition, so it is measured at its best window anywhere in the trajectory, the maximum over every placement. That endpoint is biased upward by construction. The learnable arm is measured on the same statistic, because comparing a maximum against a fixed-point value is not a comparison.

Endpoint	Learnable	Unlearnable	Ratio
best window, moving frame	`+0.1088` `[+0.1017, +0.1159]`	`+0.0043` `[+0.0034, +0.0052]`	25x
best window, frozen frame	`+0.1609` `[+0.1549, +0.1669]`	`+0.0094` `[+0.0079, +0.0108]`	17x
random-time window, moving	`-0.0019` `[-0.0232, +0.0194]`	`-0.0025` `[-0.0039, -0.0010]`	,

The unlearnable arm is below the 0.05 materiality floor in both frames, by an order of magnitude, despite being handed the most favourable endpoint available. It was given every chance to clear the floor and did not.

The kill test does not fire. The expansion requires learning.

An incidental confirmation of D6

Worth recording because it was not designed for and is a clean replication:

Learnable arm, window centred at the transition
frozen frame	`+0.0873` `[+0.0814, +0.0933]`, material
moving frame	`+0.0046` `[-0.0012, +0.0104]`, not material

A symmetric window centred on a peak cancels the rise against the fall, so in the moving frame the at-event number reads as nothing while the best window elsewhere reads +0.1088. In a frozen frame the tail energy does not unwind (D6), so the same window at the same place is material.

This is exactly D6's finding arriving unbidden in a pilot built for something else, and it is a concrete warning: a null from a symmetric window centred on an event is not evidence of no effect until the frame is stated.

Verdict

The control holds. No material expansion without learning, in either frame, on the most generous endpoint available.
The programme's central distinction survives its first direct test. "The expansion accompanies acquisition" is now separated from "the expansion accompanies training", which it had never been.
The prior was that this would not fire, and it did not. It was still worth running: this was the cheapest experiment that could have shown the central claim to be about something other than what it says, and a programme that has never run it cannot say so.

Limits

**This shows the expansion needs learning, not that it needs acquisition of this task.** An arm that memorises a fixed random labelling: learnable, but with no structure to generalise, is a different and more interesting control, and is E5's grokking question. It is not tested here.
One task, one width, one architecture, one optimizer, one rank, 8 seeds per arm.
Random targets are an extreme null. The gradient signal is pure noise, so this rules out the weakest possible alternative explanation. A partially learnable arm: noisy labels at, say, 50% corruption: would test whether the expansion scales with how much there is to learn, and is the obvious follow-up.
The unlearnable arm's best window is small but not zero (+0.0043, interval clear of zero). That is what a maximum-over-placements endpoint does to noise, and is why the endpoint was chosen to be generous rather than fair.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
batch size: How many examples the model looks at before updating itself once. Bigger batches give a steadier but more expensive update.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
held-out: Data the model was never trained on, kept back specifically to test it. Scoring a model on data it has already seen measures memorisation, not learning.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
optimizer: The algorithm that decides how to change the model given the gradient. AdamW and SGD are two common choices and they behave differently.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
vocabulary: The set of distinct symbols a model can read and produce.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

Why we do this. The same care shown above is what we apply to client work. We build custom AI models that you own and run yourself, using our own platform, and the reason we publish research like this is that a model is only worth what its evaluation is worth. If a supplier cannot show you the experiments that went against them, you cannot check the ones that went their way.

All research from this programme →