How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

A Third Was Model Size

By Joshua Bolick, Awareness Software Group ~9 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: How much model does a task need, and what changes when it has more? – The abrupt jump is what spare capacity buys -- it fades smoothly as the model shrinks, long before the model stops working. Capacity and task difficulty act separately, not as a ratio.

In plain English

What we asked. Our last result showed that part of the slowdown from a bigger vocabulary comes from the model being bigger, not the task being harder. So we measured the vocabulary effect again with every model built at the same size.

What we found. The effect shrank by about a third but stayed large. A bigger vocabulary really is harder to learn, costing about twice what its information content alone would suggest, rather than the three times we reported before.

Why it matters. Corrections like this are why we keep testing our own results: the direction held, the number was too big.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 11 training runs (one anchor) plus N19's and N6's committed runs. The design and kill test were committed (0f91ccd, docstring rewrapped in 6782e64 with no logic change) before any run.

Program v2 Bucket N, item N26. Decisive computation: analysis/vocabulary_size_matched.py. Output: analysis/vocabulary_size_matched.json. Reproduce with python analysis/vocabulary_size_matched.py (about fifteen minutes on a throttled laptop CPU); --reuse re-derives every endpoint.

The question

N19 found that a third to a half of the vocabulary's effect on learning time is model size: output slots a task never uses still slow learning. N14's free-form fit gave the vocabulary an exponent of 2.637 on log2 V, 3.12 times the lag's 0.846. N26 measures the vocabulary exponent with every model built at 128 slots, tasks V = 8, 32, 128, lag 4, widths 96 and 48, N6's five seeds; the same exponent on N6's native models (V slots for a V-symbol task) is computed beside it.

**Holding model size fixed removes about a third of the vocabulary effect.** How much longer a model takes to learn as its vocabulary grows. Grey: each model sized for its own vocabulary, as in the earlier study. Blue: every model built at the largest size, so only the task changes. The last bar is the earlier estimate. *Matching model size cuts the effect by 30 to 36 percent, but a large effect remains: a bigger vocabulary is harder to learn in its own right, less so than we first said.*

Kill test, fixed before execution: at both widths the size-matched exponent's interval contains 2.637. Anchor: the V = 32 cell reproduces N19's padded run exactly. It holds.

Result: the kill test does not fire

Width	Size-matched exponent	Native exponent
`96`	`2.037` `[1.923, 2.151]`	`2.905` `[2.777, 3.034]`
`48`	`1.686` `[1.573, 1.798]`	`2.647` `[2.559, 2.734]`

Mean steps to `0.5`	`V = 8`	`V = 32`	`V = 128`
Width `96`, `128` slots	`34`	`84`	`194`
Width `96`, native	`16`	`54`	`194`
Width `48`, `128` slots	`60`	`146`	`250`
Width `48`, native	`26`	`87`	`250`

With model size held fixed, the vocabulary exponent falls by 30% at width 96 and 36% at width 48. The native exponent at width 48, 2.647, reproduces N14's 2.637 from a different fit, which is the check that the two are measuring the same thing.

What survives of N14

N14's headline was that vocabulary costs 3.12 times what the bit-count says, measured as the ratio of the vocabulary exponent to the lag's (2.637 / 0.846). The lag does not change the model's size, so its exponent needs no correction. With size matched, the ratio is 2.0 at width 48 and 2.4 at width 96. Vocabulary still costs about twice what the bit-count says, but not three times: roughly a third of N14's excess was model size. The direction of N14's finding survives; its number does not.

What stands

Kill test does not fire. Size-matched exponents 1.686 and 2.037, both clear of 2.637.
About a third of the vocabulary exponent is model size (30%-36%), consistent with N19's d.
Vocabulary's excess over the bit-count shrinks from 3.1x to 2.0-2.4x and remains.

Limits

Three vocabularies, two widths, lag 4. The lag exponent is N14's, from native models.
At 128 slots the V = 8 task leaves 120 slots unused, the largest padding here; whether the size cost is linear in unused slots is not tested.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

exponent: The number in a power law that says how strongly one quantity responds to another. A bigger exponent means a steeper response to the same doubling.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
vocabulary: The set of distinct symbols a model can read and produce.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.