How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 22 of 39

No Part of the Model Is Special

By Joshua Bolick, Awareness Software Group ~10 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. A model's memory is a large bundle of numbers with no natural divisions. We forced some divisions on it, splitting the memory into separate compartments that were not allowed to communicate, and asked whether the reorganisation we study concentrates in one compartment. If it did, you could target that compartment: grow it, prune the others, spend your effort where it counts.

What we found. Nothing concentrates. At every split we tried, the compartments differed from each other by no more than two repeats of the same compartment differed from each other, which is another way of saying they did not differ at all. Splitting the model up did make it learn later and score slightly worse, so the connections we cut were doing real work, but no individual piece was carrying the effect.

Why it matters. This closes a line of work we had been holding open. An earlier programme of ours found that you can rearrange a model's internal coordinates almost freely without hurting it, which suggested there was nothing specific to grab hold of. The obvious objection was that coordinates are the wrong unit, and that real compartments might behave differently. They do not. The same negative result holds one level up, which makes it a stronger statement than it was.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 27 training runs, no GPU, no cost.

Program v2 tier 4, item E8. Decisive computation: analysis/modular_state.py. Output: analysis/modular_state.json. Reproduce with python analysis/modular_state.py in about forty minutes on a throttled laptop CPU.

The question

Program v1 closed on a negative: coordinate-level geometry is fungible. Reshaping which directions a model uses costs nothing, at every width, difficulty, subspace and weight tested. E8 asks the same question one level up, at the architectural level.

**Cutting a model into separate compartments singles out none of them.** We split a model's memory into separate compartments that cannot talk to each other, then measured how much each one reorganised itself at the moment the model learned the task. The left group is a split into two compartments, the right into four. The whiskers are 95% confidence intervals across five runs. No compartment stands out. Within each split, the bars sit on top of each other, and the differences between compartments are no bigger than the differences between two repeats of the same compartment. That is a negative result with a practical edge: it had been worth hoping that the thing we study concentrates somewhere specific, because then you could target it, grow it or prune it. It does not. Splitting the model up also made it learn later and score slightly worse, so the connections we cut were doing real work even though no single piece was special.

Hard-partition the hidden state into blocks with a block-diagonal recurrence, so the blocks are separate recurrent streams that share an input and a readout. Does the transition and its expansion localise to one block, or spread across all of them?

Localisation would matter. It would give structural interventions their first legitimate target, grow, prune or freeze blocks, where v1's coordinate-level version proved there was nothing to take hold of.

Kill test, from docs/research-program-v2.md: expansion is uniform across blocks at every partition tested.

How the partition is imposed, and why it needed checking

The recurrent matrices are masked to block-diagonal after every optimizer step. Two details matter and both were verified rather than assumed:

A GRU stacks its three gates in weight_hh, so the mask is applied to each gate's own square submatrix. Masking the stacked tensor would be a different and meaningless operation.
Re-imposing after every step is essential, because AdamW's update is dense. A single unmasked step leaks cross-block weight back in, and the partition would quietly decay away over training.

Verified directly: off-block entries are exactly zero, with 100% / 50% / 25% of each gate kept at 1 / 2 / 4 blocks, and the unpartitioned arm is a genuine no-op through the same code path.

Result: uniform

Partition	Between-block spread	Seed spread within one block	Ratio
2 blocks of 24	`0.0064`	`0.0360`	`0.18`
4 blocks of 12	`0.0374`	`0.0368`	`1.02`

The kill test fires. The right comparison is between-block differences against seed-to-seed variation inside a single block, because with five seeds some block is always largest and the question is whether that means anything.

At 2 blocks the answer is decisive: blocks differ by a fifth of what seeds do. Whatever separates the two blocks is far smaller than what separates two runs of the same block.
At 4 blocks the between-block spread is 1.02x the seed spread, which is to say the blocks differ by exactly as much as chance, since that ratio is the definition of no effect.

The secondary check agrees and is weaker: the largest block is the same one in 60% of runs against 25% by chance. At five seeds that is not evidence of anything, and it is reported as the lesser statistic rather than as support. The variance ratio is the one carrying the verdict.

So v1's coordinate-level result reappears at the architectural level. Partitioning the state into genuinely separate recurrent streams does not concentrate the phenomenon anywhere. Structural interventions still have nothing to grab.

Partitioning is not free

The blocks do not differ from each other, and the partition itself costs plainly:

Partition	Transition	Final accuracy	Expansion
unpartitioned	`96`	`0.9888`	`0.2261` `[0.1814, 0.2707]`
2 blocks	`117`	`0.9802`	`~0.133`
4 blocks	`134`	`0.9706`	`~0.068`

Cutting the recurrence into blocks makes the transition later and the final accuracy lower, monotonically. So the recurrence's cross-block connectivity is doing real work, which is consistent with D1 and D5 finding the recurrence load-bearing and the readout not.

The expansion column is not a controlled comparison and no claim rests on it. Rank is a fixed fraction of block dimension, which makes blocks comparable within a partition: that is what the headline uses. It does not make a 12-dimensional block comparable to a 48-dimensional one: residual energy in a small space behaves differently at a matched fraction. The monotone decrease may be real or may be dimensionality. Settling it needs a design that varies block count at fixed block size, which means varying total width.

Verdict

Expansion is uniform across blocks at both partitions tested. The kill test fires.
At 2 blocks this is decisive (ratio 0.18); at 4 blocks the ratio is 1.02, which is no-effect rather than a weak effect, though five seeds is thin for the secondary check.
Structural interventions gain no target. This is program v1's central negative, restated at a level v1 did not test.
Cross-block connectivity is load-bearing, which is a positive finding arriving sideways: partitioning delays the transition and costs accuracy.

Limits

One task, one width, five seeds, two partitions, AdamW.
The 4-block arm is underpowered for its secondary check. Distinguishing a 60% argmax consistency from 25% chance needs roughly twenty seeds, not five. The variance ratio does not have that problem and is what the verdict uses.
Blocks are equal-sized and contiguous. An unequal or interleaved partition is a different experiment, and a phenomenon localising to a set of coordinates that straddles block boundaries would be invisible to this design.
The expansion endpoint is max-minus-baseline, positive under noise by construction. Only the within-partition comparison between blocks is claimed.
Only the recurrence is partitioned; weight_ih and the readout stay dense, by design, so that a block is a separate stream rather than a relabelling of coordinates. A fully partitioned model, input and readout included, is a different and more severe intervention.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
AdamW: A widely used training algorithm that adapts how big a step it takes for each individual parameter. The default in most modern AI training.
baseline: The thing you compare against. A result without one is not a result.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
hidden state: The model's working memory: the internal numbers it carries from one step of a sequence to the next.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
optimizer: The algorithm that decides how to change the model given the gradient. AdamW and SGD are two common choices and they behave differently.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
residual energy: Our measure of how much of a model's internal activity is left over after accounting for its few strongest directions. High means the activity is spread out; low means it is concentrated.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
subspace: A slice of a model's internal space: some directions kept, the rest ignored.
variance: How spread out a set of numbers is. Rising variance in a signal means it is becoming more erratic.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

Why we do this. The same care shown above is what we apply to client work. We build custom AI models that you own and run yourself, using our own platform, and the reason we publish research like this is that a model is only worth what its evaluation is worth. If a supplier cannot show you the experiments that went against them, you cannot check the ones that went their way.

All research from this programme →