How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 40 of 40

The Lean That Carries Nothing

By Joshua Bolick, Awareness Software Group ~10 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. Two of our earlier findings pointed at the same part of a model. One said that a single direction of change carries most of what a model learns at its critical moment. The other said that this direction leans unusually hard on the connections each internal unit has to itself, which is how the model remembers anything. Put together they suggest something appealing: that those self-connections are where the learning actually happens. That would have given us the first concrete thing to intervene on in this whole project.

What we found. It is wrong. We removed each part separately during the critical moment and measured how much learning was lost. Removing everything except the self-connections cost about 35 steps. Removing the self-connection part cost about 5, which is barely more than removing a random direction of the same size. The lean is real and it carries nothing.

Why it matters. The thing that made this a fair test is worth explaining, because without it the answer would have been meaningless. The self-connection part is naturally tiny, about half a percent of the whole, while the rest is around thirteen percent. If we had simply removed each and compared, we would have been measuring which piece was bigger, not which piece mattered. So we padded the small one with extra random directions until both removed exactly the same amount, and only then compared. The general lesson is one we keep relearning: where something sits inside a model and what happens when you take it away are different questions, and it is easy to answer the first and believe you have answered the second.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 32 training runs plus calibration, no GPU, no cost.

Program v2 Bucket J, item J3. Decisive computation: analysis/diagonal_surgery.py. Output: analysis/diagonal_surgery.json. Reproduce with python analysis/diagonal_surgery.py in about forty minutes on a throttled laptop CPU.

The question

I3 measured where the leading gradient direction's mass sits and found it leans on a GRU's self-connections, 4.6x what a span-random direction from the same buffer puts there, in 20 of 24 runs. That is anatomy. It says nothing about whether the diagonal part is what carries the learning.

**The part we suspected mattered turns out to carry almost nothing.** An earlier result found that a single direction of change carries most of what a model learns at its critical moment, and a later one found that this direction leans unusually hard on the connections each internal unit has to itself. The obvious conclusion is that those self-connections are where the learning lives. Here we removed each part separately during the critical moment and measured the delay. All three arms are cut down to remove the same amount, which is essential: the self-connection part is naturally tiny, so an uncorrected comparison would just be measuring size. The obvious conclusion is wrong. Removing everything except the self-connections costs about 35 steps of learning. Removing the self-connection part costs about 5, which is barely more than removing a random direction of the same size, and it removes a quarter more than that random direction does. So the lean is real and it carries nothing. Where a direction's weight sits and what its removal costs are different questions, and we now have different answers to them.

F4 is the causal instrument: project a direction out of the gradient during a window before the transition and see what it costs. J3 points it at I3's finding by splitting the leading direction into its diagonal and off-diagonal components and removing each separately.

Kill test: removing the diagonal component costs no more than a matched off-diagonal component.

The matching is the experiment

The two components do not carry the same share of the gradient: that is exactly what I3 measured:

Component, one direction	Share of gradient energy removed
diagonal	`0.0028`
off-diagonal	`0.1338`
span-random (F4's control)	`0.0360`

Unmatched, the diagonal arm strips 0.3% of the gradient and the off-diagonal arm 13%. Any comparison of their costs would be measuring the energy, not the direction. So each arm is brought up to 0.1338 by adding span-random companion directions from the same buffer, orthogonalised against the component so nothing is double-counted: the diagonal arm needs 5, the off-diagonal arm 0, the control 3.

Result: the off-diagonal component carries it, by `6.5x`

Arm	Cost in steps	Energy removed	Resolved
off-diagonal	`+34.5` `[+32.7, +36.3]`	`0.1623`	8/8
diagonal	`+5.3` `[+4.4, +6.2]`	`0.1604`	8/8
span-random (control)	`+3.6` `[+2.8, +4.4]`	`0.1280`	8/8

Paired within run, diagonal minus off-diagonal: -29.2 [-31.7, -26.6] steps. The diagonal and off-diagonal arms are matched to within 1.2% on energy removed and differ 6.5-fold in cost.

And the diagonal component is barely distinguishable from random. It costs +5.3 against the span-random control's +3.6, while removing 25% more energy than that control. Removing the part of the leading direction that sits on the self-connections is close to removing nothing in particular.

Verdict

The kill test fires, and in the stronger direction. The diagonal component does not merely fail to cost more than a matched off-diagonal one: it costs dramatically less, and about as much as a random direction of comparable energy.
I3's enrichment is anatomy, and stays anatomy. The leading direction really does lean on the diagonal, and the diagonal is not where its load sits. Both statements are true and they are not in tension: where a direction's mass is and what its removal costs are different questions, and this programme has now measured both and got different answers.
The tempting consolidation is dead. I3 raised the possibility that F4's load-bearing direction and G4's self-connections were one mechanism, and this closes it: they are two, and the concrete intervention target J3 was written to find does not exist.
F4's result is unchanged and slightly sharper. Whatever the leading direction is doing, it does it off the diagonal.

Limits

Eight seeds, one width, one task, one architecture, one window position. The paired interval is tight ([-31.7, -26.6]) and the effect is large, so this is not the marginal case J1 was built to find, but the sample is small in absolute terms.
The energy match is good between the two components and looser for the control. The span-random arm removes 0.1280 against 0.16 for the other two, so part of its lower cost is less energy. That weakens the diagonal-versus-control comparison but not the diagonal-versus- off-diagonal one, which is the kill test.
Companion directions are not inert. The diagonal arm carries 5 span-random companions and the off-diagonal arm none, so the diagonal arm's +5.3 includes whatever those companions cost, which the control suggests is most of it. If anything this overstates the diagonal component's contribution, making the null stronger.
A component is not a mechanism. This shows the diagonal part of this direction is not load-bearing during this window. G4's finding about self-connections at matched sparsity is a different measurement and is untouched.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.