How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Unused Slots Still Cost

By Joshua Bolick, Awareness Software Group ~11 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: How much model does a task need, and what changes when it has more? – The abrupt jump is what spare capacity buys -- it fades smoothly as the model shrinks, long before the model stops working. Capacity and task difficulty act separately, not as a ratio.

In plain English

What we asked. Earlier we found that giving a model more symbols to choose from made it much slower to learn, more than the extra difficulty should explain. But more symbols also means a bigger model, because the model needs a slot for each one. So which was it?

What we found. We built the bigger model and gave it the easier task, leaving three quarters of its slots unused. It learned more slowly than the small model, landing between a third and a half of the way towards the big model on the hard task. The unused slots were never chosen, yet they still slowed learning down.

Why it matters. Two lessons. Output classes you do not need are not free. And when one change alters two things at once, separate them before crediting either.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 22 training runs (two anchors) plus N6's committed 40 runs. The design and kill test were committed (5b02b17) before any run.

Program v2 Bucket N, item N19. Decisive computation: analysis/padded_vocabulary.py. Output: analysis/padded_vocabulary.json. Reproduce with python analysis/padded_vocabulary.py (about forty minutes on a throttled laptop CPU); --reuse re-derives every endpoint from the saved series.

The question

N6 and N14 found that a larger vocabulary delays learning far more than its bit-count predicts (N14: 3.12x). But a recurrent model's embedding and readout are both vocabulary by width, so a larger vocabulary is also a larger model: at width 48, 17,280 parameters at V = 32 and 26,496 at V = 128. N19 separates the two by padding: the padded arm is N6's V = 128 model, parameter for parameter, trained on N6's V = 32 task, whose symbols use only 32 of its 128 slots.

**Output slots a task never uses still slow learning down.** Three set-ups at four model widths. The dashed line is a model built for 128 symbols trained on a task that only ever uses 32 of them, so its size matches the top line and its task matches the bottom one. *It lands in between: roughly a third to a half of what looked like 'a harder task' was really 'a bigger model'. The unused slots are never predicted, yet they slow learning by 1.5 to 2.2 times.*

**How much of the vocabulary effect is model size.** For each width, where the padded model's learning time sits between the small model and the large one, measured on a log scale, with 95% intervals from resampling the training runs. *Every width lands between a third and a half. The vocabulary effect earlier records attributed to task difficulty is partly model size, and neither explanation alone is enough.*

N6's own loop, seeds, widths 96, 64, 48, 32, 2,500 steps. Endpoint: first step at held-out accuracy 0.5. Where the padded arm falls: d = ln(padded / V32) / ln(V128 / V32) -- 0 behaves like the V = 32 model, 1 like the V = 128 model -- with an interval from resampling seeds jointly.

Kill test, fixed before execution: at every width, d's interval lies entirely above 0.5 (the effect is parameter count). Anchor: N6's V = 32 and V = 128 cells at width 48 reproduce exactly through this loop. It holds.

Result: the kill test does not fire, and neither reading is clean

Width	`V = 32`	Padded (`V = 128` model, `V = 32` task)	`V = 128`	`d`
`96`	`54`	`84`	`194`	`0.35` `[0.33, 0.36]`
`64`	`72`	`128`	`245`	`0.47` `[0.35, 0.63]`
`48`	`87`	`146`	`250`	`0.49` `[0.45, 0.55]`
`32`	`106`	`236`	`779`	`0.40` `[0.28, 0.60]`

(Mean steps to 0.5; every arm reached it on all five seeds at every width.)

The padded model sits between the two at every width: between a third and a half of the way, in log steps, from the V = 32 model to the V = 128 model. Carrying 96 unused slots slows learning by 1.5-2.2 times with the task unchanged. At width 96 the interval excludes both 0 and 0.5: there, about a third of the vocabulary's effect is model size and two thirds is difficulty; at the other widths the split is closer to half and half, and the intervals cannot tell half from a little less.

The unused slots are not what is being learned: at the end of training no held-out prediction lands on them at any width, and early on only the narrowest model predicts them at all (8% of predictions at the first evaluation at width 32).

What this does to N14 and N6

N14's 3.12x mixes two effects. Roughly 35%-50% of the vocabulary's effect on learning time, in log steps, is the larger embedding and readout rather than a harder task. Its direction -- vocabulary costs more than the bit-count says -- survives only in part, and N14 carries a banner saying so.
N6's difficulty axis is partly a size axis, for the same reason. N6's own finding, that capacity and difficulty do not collapse onto one ratio, is not changed by this; its vocabulary rungs are now known to carry both.

What stands

Kill test does not fire; the difficulty reading does not survive intact either. d is 0.35-0.49.
A larger readout slows learning on its own, by 1.5-2.2 times here, with no change to the task.
Practical reading: a vocabulary bigger than the task needs is not free. Unused output classes slow learning even though they are never predicted.

Limits

One lag (4), two vocabularies, one padding level. Padding V = 8 to 32, or the reverse direction the programme entry mentioned, was not run.
d rests on arm means, five seeds per arm.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
held-out: Data the model was never trained on, kept back specifically to test it. Scoring a model on data it has already seen measures memorisation, not learning.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
parameters: The adjustable numbers inside a model. Training is the process of setting them. Model size is usually quoted as a count of these.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
vocabulary: The set of distinct symbols a model can read and produce.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.