How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Wider Models Still Compete

By Joshua Bolick, Awareness Software Group ~10 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually happens at the moment a model learns? – It builds machinery rather than selecting it, working through the task in a reproducible order and trying a simpler wrong rule on the way. The visible training curve cannot tell you which is happening.

In plain English

What we asked. We found that teaching a small model one extra skill it does not need slows down everything it learns afterwards. The obvious explanation is that a small model is short of room, so we gave it twice as much room and ran the experiment again.

What we found. It barely changed. With twice the width, the unneeded skill still cost the other skills about a quarter to a third of their learning time. Only one skill showed any easing, and that only weakly.

Why it matters. So the competition is not simply a matter of the model being too small. A likelier explanation is that every skill being taught pulls on the same shared settings, however many of them there are. We have not yet separated the two.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, six seeds, 13 training runs of 3000 steps. The design and kill test were committed (acd91b3) before any run.

Program v2 Bucket O, item O21. Decisive computation: analysis/competition_width.py. Output: analysis/competition_width.json. Reproduce with python analysis/competition_width.py (about ninety minutes on a throttled laptop CPU); --reuse re-derives every endpoint from the saved series, and was checked to print the identical summary.

The question

O20 found that on identical inputs, teaching a width-64 model a skill nothing needs (G) slows the skills it learns afterwards by hundreds of steps. Two readings fit: capacity (a small model has limited room) and gradient (every supervised skill's loss pulls on the shared weights). Capacity competition should ease as the model grows; gradient competition need not.

A model twice as wide still pays nearly the same share for an unneeded skill. The same experiment at two model sizes: how much sooner three skills are learned when a skill nothing else needs is never taught, as a percentage of the normal time. The first bar of each pair is the smaller model, the second is twice as wide. Doubling the model's width barely changes the share: every skill still arrives about a quarter to a third sooner without the unneeded one. The competition between skills is not simply a lack of room that a bigger model removes.

O21 repeats O20 at width 128, same seeds and task, control and G-never, O12's own loop with only the width changed. Endpoint: each composite's paired change as a share of its control time, because a wider model learns everything sooner.

Kill test, fixed before execution: at width 128, F's share has an interval that overlaps width 64's mean share -- no evidence of easing, so the capacity reading loses its distinguishing prediction.

Anchor, in code: seed 9433's width-64 control, re-run through this script's width switch, reproduces O17's committed series exactly. It does.

Standing rule, noted in the pilot: a fixed-task width sweep makes the model progressively oversized (P11). Here that is the hypothesis -- does spare capacity remove competition? -- but it means any easing is about capacity relative to this task, not about scale.

Result: the kill test does not fire -- narrowly, on one skill of three

Acquisition steps at width 128 (all six seeds acquire every skill):

Arm	`B`	`G`	`D`	`A`	`F`	`C`	`E`
control	`90`	`280`	`332`	`402`	`668`	`758`	`1068`
`G-never`	`82`	--	`225`	`267`	`495`	`490`	`770`

Paired change without G, as a share of the control's time:

Skill	Width `64` (O20)	Width `128`
`C`	`-0.324` `[-0.766, +0.118]`	`-0.353` `[-0.399, -0.308]`
`F`	`-0.370` `[-0.622, -0.117]`	`-0.261` `[-0.333, -0.189]`
`E`	`-0.277` `[-0.470, -0.084]`	`-0.278` `[-0.339, -0.216]`

The kill test does not fire: F's width-128 interval [-0.333, -0.189] excludes width 64's mean (-0.370). But read the rest of the table before crediting it. The test is one-sided in a way the design did not intend: width 64's own interval ([-0.622, -0.117]) contains width 128's mean, so the two widths are not distinguishable in the other direction. And C and E do not ease at all: -0.324 against -0.353, -0.277 against -0.278.

Reading it

Doubling the width does not remove the competition. At width 128 every composite still arrives roughly a quarter to a third sooner when G is never taught, and the look-ups A and D are sped too. One skill of three shows a smaller share, with evidence that is weak in one direction and absent in the other. On this evidence the effect is closer to what gradient competition predicts than what pure capacity competition predicts, though this design cannot separate them cleanly.

The practical reading is O20's, now at two widths: an acquisition time is a property of the whole task. Adding a skill nobody needs costs every other skill a share of its time, and a model twice as wide pays nearly the same share.

What stands

Kill test does not fire, narrowly: F's share falls from -0.370 to -0.261, the only difference of the three and weak in one direction.
C and E are unchanged by width (-0.35, -0.28 at width 128).
Competition persists at width 128 at about a quarter to a third of each composite's time.

Limits

Two widths, one task, six seeds; width-64's intervals are wide because O20's acquisition times varied by hundreds of steps.
The kill test compared an interval with a mean in one direction only; a symmetric comparison would not have separated the widths. Recorded here as a design weakness, not re-scored.
Capacity and gradient competition are not separated. A test that keeps G's loss but blocks its demand on capacity, or the reverse, is the natural next step.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.