How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Most of It Is Pull

By Joshua Bolick, Awareness Software Group ~11 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually happens at the moment a model learns? – It builds machinery rather than selecting it, working through the task in a reproducible order and trying a simpler wrong rule on the way. The visible training curve cannot tell you which is happening.

In plain English

What we asked. Teaching a small model an extra skill it does not need slows everything else it learns. We have been trying to find out why: does the extra skill take up room, or does teaching it pull on the model every step? Two earlier attempts could not answer; this one used twelve models and a steadier way of measuring.

What we found. When the extra skill was taught half as often, it was still learned, but it cost the other skills about half as much for one of them and about two thirds as much for the other. If room were the cause, halving the teaching would have changed nothing.

Why it matters. So the cost is mostly the pull of training on something unneeded, and it scales with how often that thing is trained. In practice: an unneeded goal mixed into training is not free just because the model can hold it.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, twelve fresh seeds, 36 training runs of 3000 steps. The design, endpoint, bootstrap and kill test were committed (c88ae75) before any run.

Program v2 Bucket O, item O24. Decisive computation: analysis/competition_ratio_of_means.py. Output: analysis/competition_ratio_of_means.json. Reproduce with python analysis/competition_ratio_of_means.py (about two and a half hours on a throttled laptop CPU); --reuse re-derives every endpoint from the saved series, and was checked to print the identical summary.

The question

Teaching a small model a skill nothing needs (G) slows its later skills (O20), and doubling the width barely changes the share (O21). Room (G occupies capacity once learned) predicts that halving G's teaching leaves the slowdown intact; pull (G's loss acts on shared weights every step) predicts it halves. O23 asked this with a per-seed ratio that divided by a negative number on one seed. O24 asks it with twelve fresh seeds and a ratio of means.

**Teach an unneeded skill half as often and it costs the others about half as much.** Twelve small models, each trained three ways: an unneeded skill taught on every example, on half of them, or not at all. Each bar is how much of the slowdown it causes remains when it is taught half as often. If the skill slowed others by taking up room, the bar would sit near 1; if by pulling on the model while being taught, near 0.5. Both bars sit well below 1: teaching the unneeded skill less often removes a large part of what it costs the others, even though it is still learned. The competition is mostly the pull of training, not a shortage of room.

Three arms per seed, every run O12's own loop through O22's batch wrapper: full, half (G on a random 50% of sequences, learnable by O23's throwaway check) and none.

Endpoint: R = (mean F_half - mean F_none) / (mean F_full - mean F_none), interval by 10,000 seed-level bootstrap resamples (generator seeded 2026). Precondition: the denominator's interval excludes zero, and G is learned in the half arm on at least eight seeds. Both pass: denominator +1088 [+793, +1298]; G learned on 12 of 12.

Kill test, fixed before execution: R's interval includes 1.

Result: the kill test does not fire

Skill	Full `G`	Half `G`	No `G`	Denominator	Remaining share `R`	Seeds
`C`	`1897.3`	`1411.8`	`983.6`	`+914` `[+699, +1160]`	`0.469` `[0.332, 0.651]`	`11`
`F`	`2302.2`	`1981.1`	`1214.4`	`+1088` `[+793, +1298]`	`0.705` `[0.508, 0.889]`	`9`
`E`	--	--	--	--	too few seeds	`6`

Halving G's teaching removes about half of its slowdown of C and about a third of its slowdown of F. Both intervals exclude 1, so the pure room reading fails. C's interval contains 0.5, as pure pull predicts; F's lies between 0.5 and 1.

The gate that biases `F`, disclosed

F's ratio uses only seeds that acquired F in all three arms. In the full arm, three of twelve seeds never acquired F within 3000 steps -- the most slowed seeds, dropped from the average. Dropping them shrinks the denominator (the full-G slowdown) and so pushes R up. F's 0.705 is therefore an overestimate of its remaining share, by an amount this design cannot measure. That is the standing rule that an eligibility gate is a sampling decision; it is reported, not corrected after the fact.

Reading it

Most of the competition between skills here is pull: teaching an unneeded skill half as often removes half (C) or at least a third (F) of what it costs the others, even though it is still learned and still occupies whatever room it takes. With O21 -- a model twice as wide pays nearly the same share -- the thread from O17 reads consistently: every supervised skill's loss pulls on the shared weights, and that pull, not a shortage of room, is what slows the rest. A small room component for F cannot be ruled out.

What stands

Kill test does not fire. C 0.469 [0.332, 0.651], F 0.705 [0.508, 0.889]; both exclude 1.
Competition between skills is mostly pull on this task and width, with F's share biased upward by three unacquired seeds.
Practical reading: an unneeded objective in a multi-task mix costs the others roughly in proportion to how often it is trained, not merely by existing.

Limits

One task, width 64, one reduced share (50%). A second share would test proportionality directly.
F on nine seeds, E on six; the 3000-step budget was sized for O17's control and is tight for the full-G arm, which is the arm that loses seeds.
"Pull" is inferred from dose, not measured; gradient interference between skills was not computed.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

bootstrap: A way of estimating how uncertain a number is by repeatedly resampling the data you already have. Useful when the usual formulas do not apply.
eligibility gate: A rule that excludes a run from a measurement, usually because it did not improve enough for the measurement to mean anything. It is a decision about which runs are in your average, not just a safety check.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

In plain English

The question

Result: the kill test does not fire

The gate that biases F, disclosed

Reading it

What stands

Limits

Terms on this page

Read next

Get new results as we publish them

Want this measured on your data?

The gate that biases `F`, disclosed