How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 43 of 45

The Direction That Does Not Travel

By Joshua Bolick, Awareness Software Group ~16 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. When one of these small models suddenly gets good at its task, the change is not spread evenly through it. We found earlier that one particular direction of change does most of the work: block that one direction during a short window and the model takes far longer to learn, even though the direction is a small fraction of everything that is changing. That is the most important thing this project has found about how the moment of learning works. But we had only ever measured it inside the run that produced it, which leaves the obvious question open. Is that direction carrying something real about the task, something another model could use? Or is it just bookkeeping about one particular model's situation at one particular moment?

What we found. So we took the direction out of one training run and used it to push a second, completely separate run along the same path. The honest comparison, and the one that decides the answer, is a direction picked at random from the same donor run: if a borrowed direction does no more than a random one, nothing was carried across. That is what happened. The borrowed direction and the random one behave the same at every strength we tried. Measured as a share of what the receiving model's own direction does, the borrowed one delivers between three and seven percent, where we had said in advance that twenty would be the least we would call a real effect. There is a small difference between borrowed and random at the strongest setting, and we can explain it without anything having transferred: the borrowed direction happens to point slightly less sideways relative to what the receiving model was already doing, so it knocks it off course slightly less hard. It retains about eight percent of the alignment and delivers about the same share of the effect, two numbers we worked out separately that agree.

Why it matters. The part worth your time is that our first answer was the opposite one, and the cause was a missing minus sign. The mathematical procedure we use to find this direction gives an answer that is correct either way round, forwards or backwards, and it picks between them arbitrarily. That had never mattered before, because every previous experiment *subtracted* the direction, and subtracting something is the same operation whichever way round it points. This experiment *added* it, where the two are opposites. So half our runs were pushing with the model and half against it, and the average of those two things is not a measurement of anything. Nobody had needed a convention before, which is exactly why its absence was invisible. The general lesson is worth carrying: when you reuse a tool in a new way, the thing most likely to break is whatever the old way happened not to care about.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 112 training runs, no GPU, no cost.

Program v2 Bucket J, item J8. Decisive computation: analysis/direction_transplant.py. Output: analysis/direction_transplant.json. Reproduce with python analysis/direction_transplant.py, or re-derive every endpoint from the saved series with --reuse.

The question

F4 found that a single gradient direction carries 61% of the critical period for 14% of the gradient's energy. I3 found it diagonal-enriched relative to its own control; J3 found that enrichment carries nothing causally.

**A direction borrowed from another run acts like a random one.** When one of these models suddenly gets good at its task, one particular direction of change inside it does most of the work. We have measured that direction several times, but only ever inside the run that produced it. Here we took it out of one run and pushed a second, different run along it, to ask whether it carries anything portable. Below zero means the model learned sooner; above means later. The grey line is the honest comparison: a direction picked at random from the same donor run. The orange and grey lines sit on top of each other, so a direction borrowed from another run does no more than a random one taken from the same place. The blue line is the run's own direction, which does something quite different and pushes it to learn sooner. Whatever makes this direction important stays inside the run that grew it. It is bookkeeping about one model's particular situation, not knowledge that travels.

**The borrowed direction points somewhere else entirely.** A measure of how closely two directions point the same way: 1 would be identical, 0 means completely unrelated. This asks how much each of the directions we added had in common with the direction the receiving model was already moving in. The run's own direction overlaps strongly with what it was already doing, which is why adding it behaves like a bigger step rather than a shove. The borrowed direction and the random one are both close to zero, meaning both are essentially sideways pushes. The borrowed one is a little less sideways, and that alone accounts for the small residual difference between them, with no need for anything to have been transferred.

Every one of those measurements was made inside the run that produced the direction. So the obvious question had never been asked: is the direction content, or is it run-specific bookkeeping? Take donor run A's leading direction at A's own transition and add it to recipient run B's update during B's pre-transition window.

Kill test, fixed in advance: B's transition does not move, against a span-random transplant from A drawn from the same buffer.

Five arms, because the kill test alone is not interpretable

Arm	What it does	Why it is here
control	B alone	supplies B's transition and window
self-removal	project out B's own direction	positive control, F4's manipulation verbatim
self-transplant	add B's own direction back	separates "adding does nothing" from "adding this donor's does nothing"
transplant	add donor A's direction	the actual test
span-random	add a random mixing of A's same buffer	F4's standing control

Eight recipients, each with its own donor, so no result rests on one lucky direction. The window is F4's, the 40 steps before B's own transition; the transplant run shares B's seed, initialisation and data order, so it is bit-identical to the control until the window opens and every comparison is paired within seed. The magnitude is swept at 0.25 / 0.5 / 1.0 times B's own gradient norm.

Result: the kill test fires

Shift in transition step against the same seed's control. Positive is later.

Arm	scale `0.25`	scale `0.5`	scale `1.0`
self-removal (positive control)	`+30.9` `[+12.5, +49.3]`	,	,
self-transplant	`-7.6` `[-12.3, -2.9]`	`-9.3` `[-17.4, -1.2]`	`+4.0` `[-20.3, +28.3]`
transplant	`+1.4` `[+0.6, +2.3]`	`+4.7` `[+3.2, +6.2]`	`+13.9` `[+11.0, +16.7]`
span-random	`+1.7` `[+0.7, +2.7]`	`+5.7` `[+3.8, +7.6]`	`+17.6` `[+14.2, +20.9]`
transplant − span-random	`-0.3` `[-1.4, +0.8]`	`-1.0` `[-3.1, +1.1]`	`-3.7` `[-7.4, -0.1]`

The positive control passes. F4's removal delays by +30.9 steps on these same runs, so the endpoint sees a window-scale intervention it is known to be sensitive to. A null in the transplant arms is therefore about transfer, not about sensitivity.

The donor's direction behaves like a random direction from the donor. At the two smaller scales it does not separate from span-random at all. At the largest it separates by -3.7 [-7.4, -0.1] steps, an interval whose upper end is touching zero.

Sizing the separation, and why the ratio is the endpoint

An interval excluding zero is not on its own a finding here. The arms are compared at a precision where a few steps separate cleanly, and a few steps is nothing beside what the recipient's own direction does to the same run. So the endpoint is the transfer fraction: how much of the self-transplant's effect the donor's direction recovers, measured from the span-random floor both sit above, against a materiality floor of 20% fixed before the confirming run.

Scale	Transfer fraction
`0.25`	`+3.4%`
`0.5`	`+6.7%`
`1.0`	undecidable, the self-transplant effect is not itself separated from span-random here

So: at every scale where the separation can be sized, the donor's direction recovers a negligible share and does not separate from span-random. At the one scale where it does separate, the effect cannot be sized, because that is exactly the scale at which the denominator collapsed.

The mechanism: geometry, not content

The alignment diagnostic explains the sub-step differences without any transfer.

Direction added	Cosine with the recipient's own gradient
the recipient's own	`+0.5747` `[+0.4192, +0.7302]`
the donor's leading	`+0.0455` `[+0.0300, +0.0611]`
span-random from the donor	`+0.0253` `[+0.0166, +0.0340]`

The donor's direction is 1.8x better aligned with the recipient's gradient than a span-random direction is, and both are essentially orthogonal to it, roughly 13x and 23x less aligned than the recipient's own direction. Adding a vector along a run's gradient is a change of step size; adding one across it is a perturbation. Both transplant arms are perturbations, and the donor's is very slightly gentler because it is very slightly less orthogonal. 3.7 steps out of 150 is what that buys.

The donor retains 7.9% of the self-direction's alignment and delivers 3-7% of its effect. Those two numbers were computed independently and agree, which is the strongest thing in this record.

The correction that produced this record

The first run of this pilot returned the opposite verdict, from a bug in the intervention.

An SVD returns singular vectors up to a sign. That never mattered anywhere else in this programme, because every previous use projects the direction out, and removing u is identical to removing -u. Adding it is not. Without a convention, "transplant A's leading direction" meant "add it or subtract it according to an arbitrary output of LAPACK", differing per seed.

The unsigned run reported self-transplant at +105.6 [+78.3, +132.9] steps, a huge delay, because half the seeds were adding a vector that fought their own gradient. With the sign fixed along the donor's own mean gradient, the same arm accelerates by -9.3 steps. The sign was carrying the arm.

It also inflated the headline. The unsigned run cleared the materiality floor at 27.5% transfer, which was the largest number in its table and the least real: it was a ratio whose denominator had collapsed to +4.0 [-20.3, +28.3]. The guard now refuses to size a separation at any scale where the self-transplant effect is not itself determinate.

Orientation uses only the donor's information. Orienting by the recipient's gradient would leak the recipient into the transplanted object and manufacture the very alignment the experiment exists to test.

A small positive result, and why it is not an efficiency claim

The signed self-transplant arm is the complement of F4 and reproduces its logic from the other side: removing the leading direction delays the transition by +30.9 steps, and adding it accelerates by -9.3 [-17.4, -1.2]. That is the first acceleration in this programme, and it confirms the direction is genuinely load-bearing within its own run.

It is not a way to train faster, by J7's rule one record earlier: price the measurement in the same units as the benefit. Knowing the direction requires maintaining a 32-step gradient buffer and taking an SVD of a 32 x 17,280 matrix to get it. The saving is 9 steps out of 150, or 6%. The instrument costs far more than that, exactly as it did in J7.

Verdict

The kill test fires. The leading gradient direction does not carry transferable content. What makes it load-bearing stays in the run that produced it.
C3 closes. "Transition-vector transplantation" has been on the board since program v2 opened and is now answered.
The separation that exists is geometry. The donor's direction is a marginally gentler perturbation because it is marginally less orthogonal to the recipient, and the alignment ratio predicts the effect size independently.
F4's direction survives, in a narrower form. It is load-bearing, and it is local. Removing it delays and adding it accelerates, both within its own run.
A method note worth more than the result: a sign convention is part of an intervention's definition. Every previous use of this direction was sign-invariant, which is exactly why nobody had needed one, and exactly why its absence was easy to miss.

Limits

Eight recipients, eight donors, one task, one width, one learning rate, one window, three scales. The window and buffer are F4's and were not swept.
One direction, rank 1. F4 swept ranks; this did not. A rank-4 subspace might transfer where a single vector does not, and nothing here rules that out.
Donors and recipients differ only by seed. They share task, width, learning rate and architecture, so this is the easiest possible transfer and it still fails. That makes the negative stronger, but it also means nothing here speaks to transfer between genuinely different runs.
The transfer fraction is a ratio of two noisy quantities and is undecidable at one of three scales for exactly that reason. It is reported as undecidable rather than estimated.
The self-transplant acceleration is a by-product, not a preregistered endpoint, and it is reported at the scales where its interval excludes zero. It has not been re-run.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
critical period: A stretch of time during which something has to happen for development to proceed normally. Borrowed from biology, where it describes windows in which a young brain must receive certain input.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
subspace: A slice of a model's internal space: some directions kept, the rest ignored.
SVD: Singular Value Decomposition. A standard mathematical procedure that breaks a table of numbers into its independent directions, strongest first. It is how we measure internal structure throughout this work.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.