How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

The Easiest Skill Is Learned Last

By Joshua Bolick, Awareness Software Group ~13 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Part of a bigger question: What actually happens at the moment a model learns? – It builds machinery rather than selecting it, working through the task in a reproducible order and trying a simpler wrong rule on the way. The visible training curve cannot tell you which is happening.

In plain English

What we asked. A study published this year looked at large language models and found that skills which combine other skills tend to appear later in training than the skills they combine. That was read as evidence that learning follows a dependency order, like a syllabus. There is a problem with reading it that way: in almost any real task the combined skill is also simply the harder one, and models learn easier things first anyway. Both explanations predict exactly the same thing, so the observation cannot tell them apart.

What we found. So we built a task where the two explanations disagree. It has three skills, and the third can only be done by combining the first two. But we made that third skill very easy to do on its own, easier than either of the others. If models just learn easy things first, it should come first. It comes last, and takes more than nine times longer in company than it does alone. Dependency wins over difficulty.

Why it matters. Two things worth carrying away. The first is the shape of the method rather than the result: when two explanations predict the same thing, no amount of additional data separates them, and the only way forward is to construct a case where they disagree. That is cheap to do on small models and it is the sort of check a very large study cannot easily run. We are not claiming the larger study is wrong; we are showing that the question it leaves open has an answer, and how to get it. The second is an accident. One of the three skills is actually learned faster alongside the others than on its own, so the skills help each other rather than competing for room. We were not looking for that, and we would not have seen it without the comparison the main question required.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 6 seeds on a composite task plus 18 single-skill runs and a shuffled-target control, no GPU, no cost.

Program v2 Bucket P, item P3. Decisive computation: analysis/planted_dag.py. Output: analysis/planted_dag.json. Reproduce with python analysis/planted_dag.py; --reuse re-derives every endpoint.

The question

What do Language Models Learn and When? The Implicit Curriculum Hypothesis (arXiv 2604.08510) reports that across four model families from 410M to 13B, emergence orderings are strikingly consistent (rho = .81 across 45 model pairs) and composite tasks most often emerge after their component tasks. That is read as evidence of a prerequisite structure in learning.

**The easiest skill is learned last.** We built a task from three skills where one, C, can only be done by combining the other two. Grey bars show how long each skill takes to learn on its own; blue bars show how long the same skill takes when all three are being learned together. C on its own is a simple sum and is by far the quickest to pick up. C is the easiest of the three on its own and the last to arrive when they are learned together, taking more than nine times as long. If models simply learned easy things first, C would come first. It comes last, because it cannot be done until the other two work. This matters because a recent study of much larger models found that combined skills tend to appear after their parts, and read that as evidence of prerequisites; but in ordinary tasks the combined skill is also the harder one, so the two explanations are impossible to tell apart. Building a case where they disagree separates them. A surprise on the way: skill B is learned faster in company than alone, so the skills help each other rather than competing.

Correlation cannot separate "B needs A" from "B is simply harder than A". In a task built from skills the composite is almost always the harder thing, so "composites come later" is exactly what plain difficulty ordering predicts too. The two explanations are observationally identical unless you build a case where they disagree.

So we built one. Three skills are planted in one sequence task, with an explicit marker naming which skill each query slot asks for:

A: report the token at a fixed position, six steps into the sequence.
B, report the token at position zero. The longer retrieval.
C: report (a + b) mod 16, where a and b are exactly A's and B's answers. C genuinely requires both, so its planted position is last.

And C is made easy on its own. Measured in isolation its two operands are presented adjacent to the query, so no retrieval is needed and it is pure modular addition.

Kill test, fixed before execution: the acquisition order correlates with isolated difficulty at least as strongly as with the planted topological order.

What we found

Skill	Acquired in the composite task	Acquired on its own	Ratio	Planted position
`A`	`390.0` `[346.5, 433.5]`	`191.7`	`2.0x`	0
`B`	`101.7` `[89.4, 113.9]`	`166.7`	`0.6x`	1
`C`	`575.0` `[532.6, 617.4]`	`61.7`	`9.3x`	2

Shuffled-target control ends at chance on every skill (0.078, 0.081, 0.058 against a chance of 0.063), so the orderings above are orderings among skills that genuinely fire.

The kill test does not fire. Acquisition order correlates +0.5 with the planted graph and -0.5 with isolated difficulty.

The decisive observation is C, and it does not need a correlation

C is the easiest skill in isolation by a wide margin, 61.7 steps against 166.7 and 191.7, and it is acquired last in the composite, at 575.0. Plain difficulty ordering predicts C first. The prerequisite graph predicts it last. It is last, by a factor of 5.7 over the skill it depends on.

The ratio column is the cleanest way to see it. Every skill is a different job inside the composite than it is alone, and the penalty is not uniform: A pays 2.0x, C pays 9.3x, and B is actually faster in the composite than alone (0.6x). The skill that depends on the other two pays by far the largest penalty, which is what a prerequisite structure predicts and what difficulty ordering cannot produce.

With three skills a rank correlation can only take a handful of values, so +0.5 against -0.5 is a coarse instrument and is reported as a summary rather than as the finding. The finding is the table.

An unplanned positive: B is learned faster in company

B reaches criterion at 101.7 steps in the composite against 166.7 alone. Training on A and C alongside it helps, rather than competing for capacity. That was not predicted, is not what the kill test was about, and is the kind of thing an isolated-difficulty control is worth running for on its own.

What this does and does not support

It supports the implicit-curriculum reading, and supplies the control that reading needs: in a case built so that prerequisite order and difficulty order disagree, acquisition follows the prerequisites and runs against difficulty.

It does not establish that the published rho = .81 result is measuring prerequisites rather than difficulty. Those are real models on real tasks, where the two orderings almost certainly agree, and this record cannot say which one is doing the work there. What it establishes is that the two can be separated, that a task can be built where they disagree, and that on such a task the prerequisite structure wins. That is a method the larger result could adopt.

Limits

Three skills. A rank correlation over three points is coarse, which is why the ratios carry the argument. A wider DAG with five or six skills would let the correlation mean something.
The composite/isolated comparison is not perfectly matched: in isolation only one skill is scored, so the model sees the same inputs but a third of the gradient signal. Marker tokens keep the input distribution identical, which removes the larger confound, but "alone" still means fewer scored positions per batch.
C's isolation gives it adjacent operands, which is what makes it easy and is deliberate, but it means isolated C is a different computation from composite C, not the same one in easier surroundings. That is the design; it is also its main weakness.
One architecture, one width, one vocabulary, one criterion (halfway from chance to perfect).
Acquisition is read every 10 steps, so orderings separated by less than that are not resolved. The gaps here are 100 to 470 steps.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
confound: A second explanation you did not control for. If bigger models both learn faster and score higher, then 'fast learners score higher' may be entirely about size and not about speed.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
vocabulary: The set of distinct symbols a model can read and produce.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.