How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Where the Fault Was

By Joshua Bolick, Awareness Software Group ~13 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: How do we know our own results are real? – Repeatedly, the control rather than the measurement decided the result -- and several striking findings dissolved when the right comparison was finally run.

In plain English

What we asked. We recently found that two of our own experiments had measured their uncertainty over the wrong thing. Each held one trained model fixed, tried it on six different runs, and then reported how much the answer varied across those six runs. That number cannot tell you anything about how much the answer would have varied with a different model, which turned out to be the thing that mattered. The two experiments reached opposite conclusions because of it.

What we found. That is a fault in how an experiment was set up, not in its data, so we checked all 129 of our experiment scripts for the same mistake. We expected to find several more. There are none: the two we already knew about are the whole of it.

Why it matters. Why there are none is the useful part. There is a habit in this codebase of building a fresh model for every run, which makes the correct thing the easy thing without anyone having to think about it. Both flawed experiments had deliberately stepped outside that habit, because they were asking a question that needed one shared model. The risk was concentrated exactly where the design was new, which is a sharper warning than checking everything: a house habit protects you right up until the moment you have a good reason to leave it.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. No training at all. Scans committed pilots and committed records.

Program v2 Bucket R, item R15. Decisive computation: analysis/interval_unit_audit.py. Output: analysis/interval_unit_audit.json. Reproduce with python analysis/interval_unit_audit.py in about a second.

The question

R11 found that R5 and P8 both gave one shared donor to six receivers and reported an interval over receivers. That interval cannot see donor variation, and donor variation spans 31.7 steps, wider than the effect either record claimed. The two records reached opposite conclusions from it.

**One flawed comparison, and where it turned out to live.** We found that two of our own experiments had measured their uncertainty over the wrong thing: they held one trained model fixed and then reported how much the answer varied across everything else. This checks every script in the archive for the same mistake, by scanning the code and then reading each script the scan flags. The two we already knew about are the whole of it. We expected more, and the reason there are not more is worth keeping: there is a house habit that builds a fresh model for each run, and it makes the correct thing the easy thing. The two flawed experiments had both stepped outside that habit on purpose, to ask a question that needed one shared model. The risk was concentrated exactly where the design was new.

That is a design fault, not a data fault, and nothing had checked whether it was confined to those two.

Kill test, fixed before execution: fewer than 3 further records report an interval over a unit they did not vary.

Prior recorded in advance: good that there are more than three, because every pilot here computes intervals with the same helper over whatever list it happens to have, and "one donor, six receivers" is the cheapest way to run any transplant arm. The prior was wrong, and why it was wrong is the result.

Result: the kill test fires, and there are zero further cases

129 pilots and 144 records scanned. Two passes, because neither alone would be honest.

A structural scan of the code looks for the signature: an artefact built once and consumed inside a per-seed loop, in two detectable forms, a singular *_SEED constant distinct from the pilot's SEEDS tuple, and a name assigned from a training call outside a seed loop and read inside one. It returns five candidates.

A prose scan of the records looks for an arm described as sharing one artefact, catching what the structural pass cannot. It returns twelve records.

Every candidate was then classified by reading its code, and the classification is committed in the script with a one-line reason rather than asserted here.

Pilot	Verdict	Why
`donor_economics.py` (P8)	CONFIRMED	One shared donor to six receivers, interval over receivers, and the record's own-versus-shared equivalence conclusion rests on it
`donor_economics_second_task.py` (R5)	CONFIRMED	Same design at seven lengths; already corrected by banner
`optimizer_transition.py`	`DIFFERENT`	`SCREEN_SEED` picks a learning rate on a seed deliberately disjoint from the measurement seeds: selection hygiene, not a fixed artefact inside an arm
`phenomenon_map.py`	`DIFFERENT`	Same pattern, same reasoning
`shared_donor_spread.py` (R11)	`FIXED`	Trains a donor outside the loop and then loops over six donors: the repair, flagged because it has the same shape

All twelve prose hits beyond those already flagged say "one donor per receiver", "one donor each", or "one donor per seed", which is the correct pattern, stated explicitly in a Limits section.

Zero further cases. The kill test fires.

Why the prior was wrong, which is the useful part

There is a house convention and it is right by construction. Transplant pilots here build a donor as seed + DONOR_OFFSET, which gives one donor per receiver seed. An interval over seeds then varies the donor and the receiver together, so it sees exactly what it should. Every transplant pilot in the archive follows it, which is why the structural scan found so little.

P8 and R5 both departed from that convention on purpose, to build a shared arm, and the departure is precisely where the fault entered. The shared arm is a different question ("is one donor as good as many?"), it needs a fixed donor by definition, and neither record noticed that fixing the donor also removed the interval's ability to say anything about donors.

So the fault was not endemic. It entered at the single point where a record left a convention to ask a new question. That is a more precise warning than "check your intervals": the risk concentrates exactly where a design is novel, and a novel design is the one place a house convention is not protecting you.

The milder residual, reported rather than dropped

The two DIFFERENT pilots are not perfectly clean. SCREEN_SEED selects a learning rate on one held-out seed, so "arm X at its best learning rate" carries screening noise that the reported interval does not include.

It is a different and much smaller fault. The screening seed is disjoint from the measurement seeds, which is the thing it was put there to do; every arm was screened the same way, so it does not systematically favour one; and the quantity fixed is a hyperparameter rather than a trained artefact whose identity was shown to span 31.7 steps. It is named here so that "clean" is not claimed for something that was merely better.

What stands

Zero further records have R11's fault. P8 and R5 are the whole of it, and both already carry the finding.
The recorded prior was wrong and the reason is structural: seed + DONOR_OFFSET is a convention that makes the correct thing the easy thing.
The fault entered where the convention was left, which is a sharper rule than auditing everything.
Two candidates are a different, milder pattern and are reported rather than counted as clean.

Limits

A scan finds what its signature describes. A pilot could fix an artefact in a way that is neither a singular *_SEED constant nor an assignment from run( outside a loop. The prose pass exists because of that, and it is a keyword search over records, which is weaker than reading all 144.
"Reports an interval over the unit it varied" is judged by reading, not computed. The scan decides what must be read; the verdicts are mine, and they are committed in the script with reasons so a later reader can disagree with a specific one.
The audit says nothing about how large the error is in the two confirmed cases. R11 measured that for R5; P8's own-versus-shared claim is on a different task and has not been re-run, so it is uncertain rather than wrong, the same status P12 gave four fade-with-width results.
Only this repository's pilots. Nothing here checks the study configs under configs/ or the paid study bundles under results/, which use a different orchestration path.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

held-out: Data the model was never trained on, kept back specifically to test it. Scoring a model on data it has already seen measures memorisation, not learning.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
residual: How far the data sits from a proposed description of it, after the description has been fitted as well as it can be. A small residual means the description accounts for what was measured; a large one means something is missing.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.