How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Neither Half Works Alone

By Joshua Bolick, Awareness Software Group ~11 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Part of a bigger question: Which parts of a model actually matter? – The parts that move most are not the parts that matter, and no single component is required -- the model routes around every freeze. What a part is worth shows up only when you remove it.

In plain English

What we asked. A trained model's internal machinery can be described as two things: which directions it works in, and how strongly it works in each direction. Our two previous results found that handing the whole machinery to a fresh model gives it a large head start, but that summary descriptions of that machinery give nothing at all. So we split it down the middle and handed over one half at a time.

What we found. Neither half does anything. Given the trained model's directions but not its strengths, the fresh model is actually slower than if it had been left alone. A previous experiment found the strengths without the directions behave the same way. Put the two halves together and the result swings by nearly forty steps. What a trained model has built cannot be taken apart and handed over piece by piece.

Why it matters. Two things worth carrying away. The first is that the directions are not worthless, they are just not sufficient: they beat directions chosen at random by about fifteen steps, which is a real difference, and it is still not enough to help. Effects that only exist in combination are easy to miss entirely if you test one ingredient at a time and conclude nothing matters. The second is about a check we included out of habit. We added an arm whose only job was to rebuild the original result exactly, as proof the machinery was working. It failed, and it was right to: we had left out the model's biases, which are under two per cent of its numbers and turn out to carry about a third of the whole benefit. Without that check we would have reported a number a third too small and never noticed.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 6 receivers across 4 arms, no GPU, no cost.

Program v2 Bucket P, item P6, generated by P5. Decisive computation: analysis/carried_subspace.py. Output: analysis/carried_subspace.json. Reproduce with python analysis/carried_subspace.py; --reuse re-derives every endpoint.

The question

P1 found a donor's converged recurrent weights remove 35.8 steps [-42.0, -29.7] from a receiver's time to transition. P5 imposed the donor's magnitudes: its spectrum, and three other summary statistics, and reproduced none of it. A matrix factors into which directions and how much in each. P5 did the second. This does the first.

**Neither half works alone.** The internal machinery of a trained model can be split into two parts: which directions it works in, and how strongly it works in each of them. Reading left to right, a fresh model is handed more and more of a trained one: first directions picked at random, then the trained model's real directions, then its strengths as well, and finally its biases too. Red bars mean the fresh model was slower than if it had been left alone; green means faster. Each half on its own is worse than useless. The trained model's directions without its strengths leave the fresh model slower than doing nothing, and a separate experiment found the strengths without the directions do the same. Put both together and the result swings by nearly forty steps. Whatever is being handed over cannot be split into parts and handed over piecemeal. The last step is also worth noting: the biases are under two per cent of the model and carry about a third of the whole benefit.

Kill test, fixed before execution: the vectors arm does not overlap P1's -35.8 [-42.0, -29.7], or does not separate from a random orthonormal basis.

Anchors, enforced in code: P1's transplant arm and P5's all arm must both re-derive from their committed series. The script exits 2 otherwise.

What we found

Each arm imposes part of a converged donor onto the receiver's own random initialisation. Negative is sooner.

Arm	Directions	Magnitudes	Biases	Effect
`vectors-random`	random	receiver's	receiver's	`+28.3` `[+22.0, +34.7]`
`vectors`	donor's	receiver's	receiver's	`+13.3` `[+9.0, +17.6]`
`weights`	donor's	donor's	receiver's	`-23.3` `[-27.6, -19.0]`
`weights-and-biases`	donor's	donor's	donor's	`-35.8` `[-42.0, -29.7]`

The bottom row reproduces P1 exactly, which is the reconstruction sanity check and it passes.

The kill test fires. The donor's directions alone land at +13.3, nowhere near -35.8, and still a net delay against doing nothing.

But the directions are not nothing

vectors separates cleanly from vectors-random: +13.3 [+9.0, +17.6] against +28.3 [+22.0, +34.7], intervals well apart. The donor's directions are worth about fifteen steps relative to random directions of the same shape. They just are not worth anything relative to doing nothing at all.

The head start is irreducibly joint

Read against P5, the picture is unusually clean:

donor's magnitudes alone (P5's spectrum): +10.0, useless
donor's directions alone (vectors): +13.3, useless
both together (weights): -23.3, a swing of nearly 37 steps

Neither factor does anything on its own and together they do most of the work. There is no single carried property, which is why P5 found no statistic and why this record does not name one either. What transfers is a matrix, and a matrix is not the sum of its parts.

288 parameters carry a third of it

weights-and-biases beats weights by 12.5 steps, about a third of the whole effect, and the gate biases are 288 of 17,280 parameters, 1.7% of the model. P5 tested biases alone and got +5.8, nothing. So the biases contribute only in the presence of the donor's weights: another interaction, and a large one for so few numbers.

The sanity check that earned its place

The first execution's top rung stopped at weights and was asserted to reproduce P1. It gave -23.3 against P1's -35.8 and failed. The cause was not a broken reconstruction: the imposition loop skips bias tensors, while P1's arm copies them, so the two were simply different arms. Splitting them into two rungs is what made the bias contribution visible at all.

Had the sanity check been omitted, or written loosely enough to pass, the -23.3 would have been reported as P1's effect reproduced and a third of the phenomenon would have been invisible.

Limits

The transition endpoint is quantised to 5 steps, so differences smaller than that cannot be ranked. The gaps here are 12 to 37 steps.
The factorisation is per gate block of each recurrent matrix, which is the natural unit for a GRU but is a choice; a whole-matrix SVD would mix the gates.
vectors keeps the receiver's singular values in their own order, paired against the donor's vectors by rank. Other pairings are possible and are not swept.
Six receivers, one width, one task, one graft point, one donor per receiver.
Nothing here is an efficiency claim. The donor still costs 400 steps to save 36.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
parameters: The adjustable numbers inside a model. Training is the process of setting them. Model size is usually quoted as a count of these.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
singular value: A measure of how much a mathematical operation stretches things along one particular direction. The largest one describes the direction of strongest effect.
SVD: Singular Value Decomposition. A standard mathematical procedure that breaks a table of numbers into its independent directions, strongest first. It is how we measure internal structure throughout this work.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.