How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 25 of 39

It Is Not How Much You Remove

By Joshua Bolick, Awareness Software Group ~10 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. An earlier result of ours found that this kind of model has a narrow window, about forty steps before it suddenly learns, during which one part of it has to be working. Stop that part from learning during the window and the model takes roughly twice as long to get there. Here we asked how much of that damage comes from one single direction inside that part, rather than from the part as a whole.

What we found. Most of it. Removing one direction takes away fourteen percent of the training signal and causes sixty-one percent of the damage that stopping the whole part does. Removing four directions takes away considerably more and barely adds to the harm. But the striking comparison is with timing: the very same single direction, removed at a randomly chosen earlier moment, takes away three times as much of the signal and costs a third as much.

Why it matters. So it is not how much you remove that matters. It is which direction, and at which moment, together. That is worth knowing because the intuitive way to think about damaging a training process is in terms of quantity, and quantity turns out to be close to irrelevant. We are also careful to say what this does not show: one direction is not the whole story either, since removing it still costs clearly less than stopping the whole part. Both extremes are wrong, and an experiment that had only tested the two extremes would have reported whichever one it happened to include.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 48 training runs, no GPU, no cost.

Program v2 Bucket G, item G2. Decisive computation: analysis/critical_direction.py. Output: analysis/critical_direction.json. Reproduce with python analysis/critical_direction.py in about fifty minutes on a throttled laptop CPU.

The question

D5: the recurrence has a critical period. Freezing it for the forty steps before the transition costs +19.1 steps, against +8.5 for the same freeze at a random earlier time, and the cost collapses to +0.1 forty steps after.
F4: the causal weight sits almost entirely in the single leading gradient direction.

Inside D5's window, is the recurrence as a whole what must be learning, or would removing one direction from it do the same damage?

**Most of the damage comes from one direction, at one moment.** An earlier result showed this model has a forty-step window before it learns in which one part of it has to be working: stop that part learning during the window and the model takes much longer to get there. Here we remove less and less of that part to find out how much of the damage a single direction accounts for. The percentage under each bar is how much of the training signal was actually taken away. Removing one direction takes away 14% of the signal and causes 61% of the damage that stopping the whole part does. Removing four takes 23% and barely adds to it. But compare the middle bar with the one beside it: the same single direction removed at a randomly chosen earlier moment takes away three times as much of the signal and costs a third as much. So it is not how much you remove that matters, it is which direction and at which moment, together. The grey bar on the right removes an ordinary direction and does essentially nothing, which is what makes the others mean something.

Kill test: top-direction removal costs materially less than a full freeze, against a span-random control at matched rank. Then the critical period is a property of the whole component, not of any direction in it.

Result

All five arms act only on the recurrent matrices, only inside a 40-step window ending at each seed's own transition. 8 of 8 seeds usable.

Arm	Delay	Energy removed	Share of the full freeze
full freeze (D5, replicated)	`+19.1` `[+18.2, +20.1]`	100%	,
top 1	`+11.6` `[+9.6, +13.7]`	14.0%	61%
top 4	`+13.6` `[+12.6, +14.6]`	23.4%	71%
span-random 1	`+0.4` `[-0.1, +0.8]`	0.9%	2%
top 1, random placement	`+4.0` `[+2.0, +6.0]`	42.2%	21%

The full-freeze arm reproduces D5 exactly, +19.1 against D5's +19.1, on the same seeds. The setup is D5's, so the nested comparison is against the right reference.

Removing one direction costs 61% of freezing the whole component, while removing 14% of its gradient. Against a matched random direction from the same span it is +11.2 [+9.4, +13.1] clear, so this is about which direction, not about perturbation.

And placement still matters for that one direction: +11.6 targeted against +4.0 at a random earlier window, a paired difference of +7.6 [+5.6, +9.6]. D5's critical period survives at the level of a single direction.

The comparison that settles what the variable is not

Read two rows against each other:

top 1, targeted removes 14.0% of the gradient and costs +11.6.
top 1, random placement removes 42.2%, three times as much, and costs +4.0.

Removing three times more of the gradient at the wrong moment costs a third as much. Energy is not the operative variable; the conjunction of direction and timing is. That is the same shape F4 found and this record finds it again from a different angle.

And it saturates. Going from one direction to four takes 23.4% of the gradient instead of 14.0% and buys +13.6 instead of +11.6: nine percentage points more energy for two steps more delay, which is F4's rank 1-to-16 saturation appearing at smaller rank.

Verdict: partly, and the remainder is the finding

The kill test half fires. One direction costs materially less than a full freeze (-7.5 [-9.7, -5.3]), so the critical period is not carried entirely by the leading direction.

But it is not a whole-component property either. 61% of the effect from 14% of the gradient, and the direction beats a matched random one by a wide margin, so it is not distributed evenly across the component.

The honest statement is the awkward one: most of the critical period is carried by one direction, and a real remainder is not. Neither "it is the component" nor "it is the direction" is right, and a study that only ran the two extreme arms would have reported whichever one it happened to include.

What this joins up

With G1, the leading gradient direction now has an unusually complete description across three separate properties:

Property	Finding
Causal: does removing it hurt?	Yes (F4, and G2 inside the critical window)
Predictive, does watching it warn?	No: its complement does (G1)
Sufficient, is it the whole story?	No: 61% of the critical period, not 100% (G2)

Prominence, necessity, predictiveness and sufficiency are four different properties of the same object, and this programme has now measured all four and found they do not agree.

Limits

One task, one width, one optimizer, eight seeds, one window length, one component.
The direction basis is built from recent recurrent-group gradients and from gradients the intervention has not touched, so "the top direction" means the direction training was going in rather than the one left over.
Energy removed differs between the targeted and random-placement arms (14.0% against 42.2%) because the gradient is more concentrated earlier in training. That difference is reported and is what makes the timing comparison informative, but it does mean those two arms are not matched on energy, only on rank and on window length.
The 39% that one direction does not explain is uncharacterised. Whether it is spread evenly across the remaining directions, or concentrated in a second small set, is the obvious follow-up and would need a rank sweep inside the window rather than the two ranks tested here.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

critical period: A stretch of time during which something has to happen for development to proceed normally. Borrowed from biology, where it describes windows in which a young brain must receive certain input.
freezing: Stopping one part of a model from changing while the rest carries on learning. A way of asking what that part contributes, by taking it away.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
optimizer: The algorithm that decides how to change the model given the gradient. AdamW and SGD are two common choices and they behave differently.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

Why we do this. The same care shown above is what we apply to client work. We build custom AI models that you own and run yourself, using our own platform, and the reason we publish research like this is that a model is only worth what its evaluation is worth. If a supplier cannot show you the experiments that went against them, you cannot check the ones that went their way.

All research from this programme →