How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 27 of 39

How You Cut a Model Matters, Not Just How Much

By Joshua Bolick, Awareness Software Group ~11 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. A model's memory is a grid of connections between its internal units. An earlier experiment showed that cutting a lot of them out costs real performance. This asks a follow-up that sounds pedantic and is not: does the damage come from how many connections you remove, or from which ones?

What we found. From which ones, clearly. Removing three quarters of the connections in tidy square groups costs about twenty fewer steps to learn the task than removing exactly the same number at random, and ends five percentage points more accurate. Structure is worth something even at identical cost.

Why it matters. The part worth reading is what nearly went wrong. Tidy groups happen to preserve every unit's connection to itself, and random cutting does not, and a unit's connection to itself is how it remembers anything. So the whole advantage might have been that one detail rather than the grouping. We only noticed while writing up the limitations. Testing it took an extra ten runs and showed the detail explains roughly half the effect, leaving the other half genuinely to the grouping. Without that check we would have claimed all of it.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 25 training runs, no GPU, no cost.

Program v2 Bucket G, item G4. Decisive computation: analysis/sparse_recurrence.py. Output: analysis/sparse_recurrence.json. Reproduce with python analysis/sparse_recurrence.py in about forty minutes on a throttled laptop CPU.

The question

E8 partitioned the hidden state into blocks and found no block is special, but also that partitioning is not free: the transition moved 96 → 117 → 134 and accuracy fell 0.9888 → 0.9802 → 0.9706 as blocks doubled. Something about cutting those connections cost real performance even though no single block carried the phenomenon.

**How you cut a model matters, not just how much.** A model's memory is a grid of connections between its internal units. We deleted three quarters of them, three different ways, always keeping exactly the same number. Green is the untouched model for reference. Blue keeps them in tidy square groups. Purple picks at random but always spares each unit's connection to itself. Pink picks purely at random. Whiskers are 95% confidence intervals over five runs. Identical amounts removed, clearly different outcomes. Cutting in tidy groups costs about twenty fewer steps than cutting at random. Roughly half of that advantage turns out to be one specific thing, keeping each unit's link to itself, which the purple bar isolates. The other half is the grouping itself. We nearly published the whole gap as an effect of grouping: the purple bar exists because writing up the limitations surfaced the alternative, and testing it was cheaper than noting it.

A block-diagonal mask confounds two things: how many connections remain, and where they are. At two blocks half the recurrent weights are gone; at four blocks, three quarters. E8 cannot say which produced the cost.

Kill test: random sparsity and block-diagonal cost the same at matched density. Then the loss is about how many connections survive, not their arrangement.

The positive control holds almost exactly

The block arms are E8 replicated, so they double as a check that this harness matches the one E8 ran on:

Arm	G4 transition / accuracy	E8 transition / accuracy
dense	`96` / `0.9880`	`95` / `0.9889`
block, 50%	`117` / `0.9828`	`117` / `0.9801`
block, 25%	`133` / `0.9746`	`134` / `0.9706`

Within one step and four thousandths of accuracy. Whatever the comparison below says, it is not saying it from a different setup.

The confound this record nearly shipped with

The first version of this pilot compared block-diagonal against plain random sparsity, matched on the number of surviving weights. That comparison is confounded, and writing the Limits section is what surfaced it:

Mask	Entries kept	Self-connections kept
block, 25%	`576`	48 / 48
random, 25%	`576`	12 / 48

The diagonal of a recurrent matrix is how a unit retains its own state across time. A block mask keeps all of it by construction; a random mask keeps roughly its density. So "block beats random" could have been entirely "keeping self-connections beats not keeping them", which has nothing to do with grouping.

So a third arm: random with every self-connection kept, the remaining budget spent at random off the diagonal, matched on total count. All three arms keep identical entry counts, and this one matches block on the diagonal too, leaving grouping as the only thing that differs.

Result: the kill test fires, and grouping survives the control

Masks are matched exactly on surviving weights, verified directly rather than assumed.

Density	Arrangement	Transition	Final accuracy
100%	dense	`96` `[92, 100]`	`0.9880`
50%	block	`117` `[111, 122]`	`0.9828`
50%	random	`125` `[117, 133]`	`0.9699`
50%	rand + diagonal	`121` `[119, 123]`	`0.9683`
25%	block	`133` `[129, 137]`	`0.9746` `[0.9713, 0.9778]`
25%	random	`154` `[140, 167]`	`0.9234` `[0.9116, 0.9351]`
25%	rand + diagonal	`146` `[143, 148]`	`0.9078` `[0.8770, 0.9387]`

At 25% density block-diagonal is better than both random arms. It transitions 21 steps before plain random and 13 steps before random-with-the-diagonal-kept, with the accuracy intervals separating in both comparisons.

How much was the diagonal?

Density	Block's total advantage	Remaining once the diagonal is matched	Diagonal explains
50%	`-8` steps	`-4` steps	50%
25%	`-21` steps	`-13` steps	38%

Roughly half the effect was the confound, and roughly half was not. Had the control not been run, this record would have claimed the whole 21 steps for grouping. Had the control been run and closed the gap, the headline would have been the opposite. It did neither, which is the outcome that needed the arm to distinguish.

At 50% the block-against-plain-random gap is not material on its own (overlapping intervals), while the block-against-diagonal gap is. That is not a contradiction: the diagonal-preserving arm has much tighter intervals ([119, 123] against [117, 133]), because fixing the diagonal removes a large source of variation between random draws.

What this changes about E8

E8's cost is not a general property of removing that many connections. It is specific to the arrangement E8 tested, and that arrangement turns out to be the favourable one.

The stated prior for this item was "moderate-high that density is what matters". That was wrong, and wrong in the more interesting direction: structure beats randomness at matched cost, and it does so even once self-retention is equalised. A recurrence cut into clean blocks keeps more of its capability than one cut at random with the same number of weights and the same diagonal.

A reading consistent with E8 and G4 together. E8 found no block is special, and G4 finds the block structure is worth something. Those are compatible: no individual block carries the phenomenon, and yet organising the surviving connections into coherent groups preserves more than scattering them. What is being preserved is the ability of a group of units to talk to each other, which is a property of the arrangement rather than of any block in it.

That also sharpens E8's negative. Structural interventions still have nothing to grab in the sense E8 tested: you cannot pick a block to grow or prune, because none is special. But arrangement is not irrelevant, so a structural intervention that changes how connections are grouped rather than which block to target is not ruled out by either record.

Verdict

The kill test fires. Block-diagonal and random sparsity do not cost the same at matched density.
Block-diagonal is better, by 21 steps and 5.1 accuracy points at 25% against plain random.
Grouping survives the diagonal control. Block still beats random-with-every-self-connection- kept by 13 steps at 25%, so the advantage is not merely self-retention.
Self-retention is 38-50% of it, which is the part this record would have wrongly claimed for grouping without the extra arm.
E8's cost is arrangement-specific, and E8 happened to test the favourable arrangement.

Limits

One task, one width, one optimizer, five seeds, two densities.
The comparison is by interval overlap rather than paired, because the two masks are different objects and the seeds are not matched arms of a single experiment. That is the weaker test, and it is the honest one here.
Random masks are drawn per seed, so the result does not rest on one unlucky arrangement, but five draws is still a small sample of the space of random masks. A block mask is one specific arrangement out of many structured ones, and this record does not test whether other structures (banded, hierarchical, small-world) behave like blocks or like random.
Only the recurrence is masked; the input and readout stay dense, following E8.
The largest alternative explanation was tested and is only half the story. Preserving every self-connection accounts for 38-50% of block's advantage; the remainder is grouping. What grouping is doing is still an account rather than a measurement, and separating it further, for example from the fact that a block mask guarantees every unit has some surviving partners while a random mask can isolate one entirely, would need another arm.
The diagonal-preserving arm has notably tighter intervals than plain random, which is expected and worth stating: fixing the diagonal removes a large source of variation between random draws.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
confound: A second explanation you did not control for. If bigger models both learn faster and score higher, then 'fast learners score higher' may be entirely about size and not about speed.
hidden state: The model's working memory: the internal numbers it carries from one step of a sequence to the next.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
optimizer: The algorithm that decides how to change the model given the gradient. AdamW and SGD are two common choices and they behave differently.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

Why we do this. The same care shown above is what we apply to client work. We build custom AI models that you own and run yourself, using our own platform, and the reason we publish research like this is that a model is only worth what its evaluation is worth. If a supplier cannot show you the experiments that went against them, you cannot check the ones that went their way.

All research from this programme →