How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 53 of 67

It Tries The Wrong Rule First

By Joshua Bolick, Awareness Software Group ~12 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. Accuracy tells you how often a model is right. It tells you nothing about what the model is actually doing. So we compared the model's answers, at every point in training, against what a handful of very simple rules would have said -- not against the correct answer, against each rule's own guess, right or wrong. The rules are the sort of thing predictive text ran on decades ago: repeat whatever word appeared a fixed number of places back.

What we found. Early in training the model's answers look most like the wrong rule, the one that just repeats the previous word. Then it switches, and from about a quarter of the way through it behaves like the correct rule instead. Every one of the five runs does the same thing in the same order. That is interesting because it is the strategy the older systems used deliberately: start with the shortest amount of context and reach further back only when you have to. The model was not told to do it.

Why it matters. There are two things worth carrying away. First, a model that has essentially solved the task still makes its remaining mistakes in one specific way -- more than three quarters of them are the answer taken from one position too near, against a control saying almost none should be. Mistakes at the end of training are not noise; they have a shape. Second, the check that mattered most here was making sure the question could be answered at all. One of the simple rules is exactly correct on this task, so agreeing with it is the same as being right, and comparing those two things would have been comparing a measurement with itself. Our first version of the analysis fell into precisely that and reported nothing.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 10 training runs, no GPU, no cost.

Program v2 Bucket M, item M2. Decisive computation: analysis/algorithm_agreement.py. Output: analysis/algorithm_agreement.json. Reproduce with python analysis/algorithm_agreement.py, or re-derive every endpoint with --reuse.

The question

Accuracy says how often a model is right. It says nothing about what it is doing. This asks a different question of the same runs: at each step of training, which of a set of named classical algorithms does the network's output most resemble: not the truth, the algorithm's own prediction, right or wrong.

**Which simple rule the network's own answers resemble, as it learns.** Instead of asking how often the network is right, this asks what it is doing: at every point in training we compare its answers to what two very simple rules would have said, whether or not those rules are correct. One repeats the word just before; the other repeats the word three places back, which is the right answer for this task. Early on the network's answers look most like the wrong rule -- it is repeating the previous word. Then it switches, and from about a quarter of the way through it behaves like the correct rule instead. So the network visits a simpler wrong method before finding the right one, which is exactly what predictive-text systems from the 1980s were designed to do deliberately. Its leftover mistakes are also systematic: more than three quarters of them are the answer from one position too near.

The candidates are imported from M1 rather than reimplemented, so "the cache model" means exactly the same thing in both records: a constant predictor, cache-1 through cache-8, and on the dispatch task cache-dispatch.

Kill test, fixed in advance: agreement with the classical model rises smoothly in step with accuracy, rather than jumping at the transition.

First, the kill test had to be shown to be answerable

M1 established that a fitted cache model scores 1.0000 on delayed-copy. Agreeing with that candidate is therefore the same thing as being correct, and comparing the two curves would be comparing a quantity with itself.

But the reported agreement is the best over all candidates at each step, and early in training that is usually a different algorithm. The first version of this pilot checked only the final point, declared both tasks degenerate, and reported nothing. Checked across the whole trajectory, the largest gap between the agreement and accuracy curves is 0.25–0.30 on delayed-copy and 0.16–0.18 on dispatch-copy-8lag. Neither is degenerate, and both kill tests are answerable.

Result: the two tasks give opposite answers

	`delayed-copy`	`dispatch-copy-8lag`
accuracy midpoint	`86.0` `[83.2, 88.8]`	`174.0` `[162.9, 185.1]`
sharpness of accuracy	`3.49` `[3.13, 3.85]`	`6.33` `[5.91, 6.74]`
sharpness of agreement	`3.87` `[3.52, 4.21]`	`4.54` `[3.77, 5.32]`
kill test	does not fire	FIRES

On the simple task, resemblance to a named algorithm jumps at least as sharply as accuracy does. On the harder task it rises more smoothly than accuracy, so the model's behaviour becomes describable by the algorithm gradually while its score jumps.

The model tries the wrong algorithm first

The identity of the best-matching candidate, by stage of training, is the same in all five seeds:

task	10%	25%	50%	100%
`delayed-copy`	`cache-1`	`cache-3`	`cache-3`	`cache-3`
`dispatch-copy-8lag`	`cache-dispatch`	`cache-dispatch`	`cache-dispatch`	`cache-dispatch`

On delayed-copy the model passes through a wrong algorithm before finding the right one. Early in training it looks most like "repeat the previous token"; by a quarter of the way through it looks most like "repeat the token three back", which is correct. That is a behavioural trajectory through algorithm space, and it is the shape Katz's backoff would predict: use the shortest context first.

On the dispatch task there is no such phase. The model resembles the correct algorithm more than any fixed-lag one from the earliest reading, long before it can execute it.

When it is wrong, it is wrong in a specific way

Among the positions the model gets wrong, how often does it predict exactly what a named algorithm would? Each figure is above that candidate's own shuffled control, the same predictions permuted across sequences, which gives the match rate expected from the marginal distributions alone.

task	the algorithm its errors match	lift above shuffled control	at step	errors still open
`delayed-copy`	`cache-2`	`+0.764` `[+0.727, +0.800]`	`241` `[158, 324]`	`2.7%`–`12.6%`
`dispatch-copy-8lag`	`cache-1`	`+0.271` `[+0.259, +0.284]`	`20` in all 5 runs	`88%`

A model that has essentially solved delayed copy makes its residual errors by reading one position too near, cache-2 where the answer is at cache-3. Over three quarters of its errors are exactly that, against a control that says almost none of them should be. The failure mode is a phase error in a periodic structure, not noise.

The denominator was guarded in advance. Once a model is solving the task only a handful of positions are wrong, so this rate is read only at steps where at least 2% of scored positions are still errors; the lowest peak in any run sits at 2.7%. On the dispatch task the peak is at step 20 with 88% of answers still wrong, so there is no denominator question there at all, and that peak precedes the accuracy jump in 5/5 runs.

Verdict

The kill test does not fire on delayed-copy (agreement 3.87 against accuracy 3.49) and fires on dispatch-copy-8lag (agreement 4.54 against accuracy 6.33). Which answer you get depends on the task.
On the simple task the model visits a wrong algorithm first, cache-1 before cache-3, in all five seeds.
Its residual errors are cache-2, +0.764 above the shuffled control: a solved model fails by reading one position too near.
Neither task is degenerate, and establishing that was a precondition rather than an observation.
Both anchors reproduced.

Limits

Five seeds per task, two tasks, one width, one learning rate. The opposite verdicts on two tasks are the result; two tasks is not a survey.
"Best over candidates" is an envelope, so its sharpness is not the sharpness of any single algorithm's agreement curve. A per-candidate analysis would answer a related but different question.
The candidate set is small and hand-chosen. A model resembling none of them would still report a best match; the shuffled control bounds how much of that is structural, but it does not supply the missing candidate.
cache-dispatch is handed the task's structure, as in M1, so its dominance on the dispatch task says the model resembles the right algorithm more than the wrong ones, not that it resembles it well in absolute terms. At step 20 it is wrong 88% of the time.
Error-conditioned agreement is measured on a shrinking set. The guard fixes the worst of it; it does not make the late-training number as reliable as the early one.
Sharpness here is comparable within this record only, for the reason M5 states: the rescaling depends on run length and sampling interval.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.