How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 19 of 39

The Strongest Part of the Push

By Joshua Bolick, Awareness Software Group ~10 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. At every step of training, a model is given a push telling it how to change. That push can be broken into directions, and a few of them carry far more of the force than the rest. We already knew that. What we did not know is whether those dominant directions are the ones doing the useful work, or just the loudest.

What we found. They are the ones doing the work, and it is almost entirely the single strongest one. Deleting just that direction, which carries about 14% of the force, made the model take 35 extra steps to learn. Deleting sixteen ordinary directions carrying 51% of the force, nearly four times as much, cost only 16. Deleting fifteen more on top of the strongest one added almost nothing.

Why it matters. The result matters less than the way we had to get it. The comparison the plan called for turned out to be meaningless: deleting random directions from a signal with seventeen thousand of them removes essentially none of the force and does nothing at all, so beating it would only have shown that doing something beats doing nothing. We had to build a fairer comparison before the question could be asked. It is also a useful counterweight to two earlier results of ours, which both found that the busiest part of a model is not the important part. Here the loudest thing really is the important thing, so the honest rule is that you cannot tell without testing.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 42 training runs, no GPU, no cost.

Program v2 tier 4, item F4. Decisive computation: analysis/gradient_surgery.py. Output: analysis/gradient_surgery.json. Reproduce with python analysis/gradient_surgery.py in about thirty minutes on a throttled laptop CPU.

The question

Program v1's K4 found that gradients concentrate in a small top subspace. That is an observation. It says nothing about whether those directions are the ones doing the work, and this project has twice been caught assuming that what moves most matters most, D1 found the busiest parameter group is the one that costs nothing to freeze, and A11 found a signal in a component the model does not need.

**Removing more of the push hurts less, if it is not the strongest part.** At each training step the model is given a push telling it how to change. That push can be split into directions, some carrying much more of the force than others. Here we deleted some of those directions during the forty steps before the model learned, and measured how much later it got there. Pink deletes the strongest directions; blue deletes the same number from the same region but chosen at random; grey deletes directions from anywhere, which turns out to remove almost none of the force at all. Compare the first pink bar with the last blue one. Deleting the single strongest direction removes 14% of the force and costs 35 steps. Deleting sixteen ordinary directions removes 51% of the force, nearly four times as much, and costs 16. So the damage does not follow how much you remove; it follows whether you removed the strongest part. Note also that the pink bars barely grow: taking the top direction does almost all of the harm, and taking fifteen more adds about eight percent.

So make it causal and time-resolved. During the forty steps before the transition, the window D5 showed is the one that matters, project the top-k directions out of the gradient and let training continue with what is left.

Kill test, from docs/research-program-v2.md: delays match the random-subspace arm. Gradient concentration is then descriptive, not load-bearing.

The backlog's control is a no-op

The backlog specifies a random-k arm matched on the number of directions removed. That control does nothing at all, and the smoke test showed it before the full run:

Arm	Energy removed	Delay
random, `k`=1	`0.0%`	`+0.0` `[+0.0, +0.0]`
random, `k`=4	`0.0%`	`+0.2` `[-0.3, +0.6]`
random, `k`=16	`0.1%`	`+0.0` `[+0.0, +0.0]`

Removing sixteen random directions from a ~17,000-dimensional gradient takes out a tenth of a percent of its energy and delays the transition by nothing measurable. Beating it would establish only that perturbing beats not perturbing, which D3 settled at about 21 steps, and the top-k arm's delay lands in exactly that region.

So a third arm does the real work. span removes k random directions drawn from inside the span of the recent gradients. The gradient lives mostly in that span, so these remove a comparable share of its energy while being in no way the top directions. top versus span is the comparison that isolates the question.

Result: the top directions are causal

Arm	`k`	Energy removed	Delay
top	1	`14.0%`	`+35.5` `[+34.1, +36.9]`
span	1	`3.1%`	`+1.2` `[+0.7, +1.6]`
top	4	`71.9%`	`+41.2` `[+37.3, +45.0]`
span	4	`12.4%`	`+3.3` `[+1.6, +5.0]`
top	16	`93.6%`	`+38.3` `[+37.5, +39.2]`
span	16	`51.4%`	`+16.2` `[+14.0, +18.3]`

Paired top-minus-span differences are material at every rank: +34.3 [+32.9, +35.8], +37.8 [+33.7, +42.0], +22.2 [+19.6, +24.8]. The kill test does not fire.

The comparison that settles it needs no ratio

Read two rows against each other:

top, k=1 removes 14.0% of the gradient's energy and costs +35.5 steps.
span, k=16 removes 51.4%, nearly four times as much, and costs +16.2.

Removing much more of the gradient hurts much less, when what you remove is not the top direction. Energy is not the operative variable. The leading direction is.

And it is essentially all in the first direction

From	To	Energy removed	Delay
top `k`=1	top `k`=16	x6.7	x1.08

Going from removing one direction to removing sixteen takes out nearly seven times the energy and buys an eight percent longer delay. The response saturates, which is D3's discontinuous-then-flat shape appearing in a third place: a step, not a dial. Whatever the top direction is carrying, the second through sixteenth add almost nothing to it.

So the backlog's framing, does the transition delay in proportion to the energy removed?, has a clean answer: no. It delays in proportion to whether the leading direction was taken.

What this does and does not overturn

D1 and A11 both found that prominence does not imply necessity, and it would be easy to read those as a general rule. F4 is the counter-example: here the dominant thing is the causal thing. The rule that survives all three is narrower and more useful, prominence and necessity are separate questions, and the only way to know which you have is to intervene. Sometimes they agree.

Consequence: low-rank gradient-compression ideas keep their mechanism at this scale. A method that preserves the top gradient direction and discards the rest is discarding something that, per the saturation result, is worth about 8% of the effect.

Limits

One architecture, one task, one width, one learning rate, six seeds, and AdamW only. D8 showed timing claims can belong to AdamW alone, and this is a timing claim.
The top subspace is estimated from a 32-step buffer of recent gradients, so "top" means recently dominant, not dominant in any global sense. A longer buffer is a better estimate and a staler one.
Rank must stay below the buffer size. At k equal to the buffer, the top directions span the entire row space and the projection zeroes the gradient outright, total ablation wearing subspace surgery's clothes. The smoke test hit exactly this (100.0% energy removed at k=16 with a 16-row buffer) and the script now asserts against it.
Projection happens before gradient clipping and the optimizer step, so AdamW's moment estimates see the projected gradient. A design projecting after the optimizer's update would be a different experiment.
Every arm still learns the task 6/6. This is a timing result, not a damage result.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

AdamW: A widely used training algorithm that adapts how big a step it takes for each individual parameter. The default in most modern AI training.
architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
optimizer: The algorithm that decides how to change the model given the gradient. AdamW and SGD are two common choices and they behave differently.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
subspace: A slice of a model's internal space: some directions kept, the rest ignored.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

Why we do this. The same care shown above is what we apply to client work. We build custom AI models that you own and run yourself, using our own platform, and the reason we publish research like this is that a model is only worth what its evaluation is worth. If a supplier cannot show you the experiments that went against them, you cannot check the ones that went their way.

All research from this programme →