How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

It Already Knew The Order

By Joshua Bolick, Awareness Software Group ~11 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. Our previous result showed that three related skills are learned in a fixed order, because the third can only be done once the first two are working. That is the kind of finding that suggests an obvious improvement: if there is a right order, teach in that order and the model should get there sooner. This is the experiment that tried it.

What we found. It does not help. Teaching everything at once is the fastest route to a model that can do all three, and deliberately staging the lessons in the correct order is slightly slower. The reason is almost funny: left to itself, the model already learns the first two skills before the third. Staging the lessons only removes a choice it was making correctly anyway. Teaching in the wrong order, though, is genuinely damaging: models given the dependent skill first frequently never got there at all.

Why it matters. Two things are worth taking from this. The first is that a real structure and a useful technique are different things, and the gap between them is where most efficiency ideas die. The dependency order here is genuine, measurable, and causally important, and it is still not something worth acting on. We have now found that same shape several times. The second is about the comparison we nearly did not include. We tested a randomly chosen teaching order as a control, and its result ranged enormously from run to run, because some random orders happen to be the right one and others the wrong one. Reported as a single average it would have looked like a mildly worse schedule rather than a coin flip. The spread was the finding, not the average.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 4 arms x 6 seeds, no GPU, no cost.

Program v2 Bucket P, item P10, generated by P3. Decisive computation: analysis/graph_curriculum.py. Output: analysis/graph_curriculum.json. Reproduce with python analysis/graph_curriculum.py; --reuse re-derives every endpoint.

The question

P3 found that acquisition follows a planted dependency graph rather than isolated difficulty, and, unplanned: that one skill is learned faster in company than alone. If learning has a dependency order and the skills help each other, the efficiency question is immediate: to get one model doing all three skills, is teaching them in dependency order cheaper than teaching everything at once?

**It already knew the order.** A companion result showed these three skills are learned in a fixed order, because the third can only be done once the first two work. The obvious next question is whether teaching them in that order saves time. Each bar is a different teaching strategy, measured by how long until the model can do all three. A fourth strategy, teaching the dependent skill first, often never finished at all and is not shown. Teaching everything at once is the fastest, and deliberately teaching in the right order is slightly slower. The reason is simple once you see it: left alone, the model already learns the first two skills before the third. Staging the lessons only takes away a choice it was already making correctly. The order is real, and it is not something you need to impose. Teaching in the wrong order, on the other hand, is genuinely damaging, so the structure does matter, just not as a lesson plan.

Choosing prerequisite concepts is not what E6's data-selection negative was about, so this was genuinely open.

Kill test, fixed before execution: graph-order does not reach all-three-acquired sooner than joint, or does not separate from reverse-order and random-order.

Anchor enforced in code: the joint arm must reproduce P3's acquisition steps. It does, to the decimal: A at 390.0, B at 101.7, C at 575.0.

What we found

Steps until all three skills are acquired, so an arm cannot win by finishing two of them early. Compute is matched: every arm sees the same batch size and the same number of scored positions.

Arm	Steps to all three
`joint`, everything from step 0	`575.0` `[532.6, 617.4]`
`graph-order`: `A`,`B` first, then add `C`	`713.3` `[542.4, 884.2]`
`random-order`, a random stage order per seed	`820.0` `[306.6, 1333.4]`
`reverse-order`: `C` first, then `A`,`B`	never, in at least half the seeds

The kill test fires. Teaching in dependency order is not faster than teaching everything at once. Its interval overlaps joint's, so it is not reliably worse either: it simply buys nothing, at a point estimate 138 steps behind.

The dependency structure is real; it is just not a curriculum

reverse-order, teaching the composite skill first, frequently never gets there at all. That is P3's finding showing up causally: C genuinely cannot be done before A and B work, so spending the early budget on it is not merely inefficient but sometimes fatal within the run length tested.

So the graph is doing real work. What it does not do is give you a better teaching order than "everything at once". Joint training already respects the dependency, because the model learns A and B first by itself, P3 measured exactly that. Staging the curriculum only removes the model's freedom to choose, and the model was already choosing correctly.

That is the whole result in one line: the model does not need to be taught the order it was going to discover anyway.

The random arm's interval is the other warning

random-order spans [306.6, 1333.4]. Some random orders happen to be the graph order and do fine; others are the reverse and nearly fail. **A staging strategy's variance is a cost too**, and an average would have hidden it: reported as a mean alone, 820.0 looks like a mildly worse schedule rather than a coin flip.

What this closes

The curriculum branch of this bucket. O6's follow-on asked whether curriculum ordering on a recovered graph reduces compute; on this task, it does not. Combined with P3, the honest summary is that the dependency structure is measurable and real, and is not an efficiency lever, which is the same shape as this programme's other efficiency findings, where the phenomenon is genuine and the ordinary alternative is already as good.

Limits

One task, three skills, one graph. A deeper or wider dependency structure, where the model's own discovered order is not the graph order: might behave differently, and that is the version worth running before treating this as general.
The stage-advance rule is "when every skill in the stage is acquired", which is an oracle: it reads the acquisition criterion the endpoint is defined by. A practical curriculum could not do this, so the staged arms are being helped here and still do not win.
reverse-order is scored as "never" within 1400 steps. It may finish later; the run length is the limit, not a proof of impossibility.
Per the standing rule this result is not width-swept, so it holds at width 64 and is not claimed beyond it. Given P9, that caveat is load-bearing rather than routine.
The transition criterion is halfway from chance to perfect, read every 10 steps.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

batch size: How many examples the model looks at before updating itself once. Bigger batches give a steadier but more expensive update.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
variance: How spread out a set of numbers is. Rising variance in a signal means it is becoming more erratic.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.