How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 43 of 68

The First Piece Is The Load-Bearing One

By Joshua Bolick, Awareness Software Group ~12 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. The exercise our models learn repeats a short pattern, and they do not learn the whole pattern at once. They learn it one piece at a time, in the same order every run. We had separately found that taking a piece away from a model during early training costs it extra rounds, and that how much it costs depends on which piece. What we could not say was which piece would turn out to be the expensive one.

What we found. The obvious guess is the piece the model learns last, on the grounds that it is the one it struggles with longest. It is the opposite. The piece the model learns first is the most expensive to take away, and the piece it learns last is the cheapest. The likely reason is that the first piece is what the rest are built on, so removing it holds up everything that follows -- but that is our reading of the pattern, not something this experiment proves.

Why it matters. Two things about how we report this. The kill test we set ourselves was whether the cost is related to the learning order at all, and it is, so by the letter the test passed -- but the idea that motivated the test was wrong, and we say both. And we are careful about size: the two early pieces clearly cost more to remove than the two late ones, but the two early pieces cannot be told apart from each other, so this is a split into two groups rather than a neat ranking of four. If there is a practical thought here, it is that the part of a job a system picks up quickest may be the part you can least afford to disturb, which is not where most people would look first.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 30 training runs, no GPU, no cost.

Program v2 Bucket N, item N13. Decisive computation: analysis/cost_and_acquisition_order.py. Output: analysis/cost_and_acquisition_order.json. Reproduce with python analysis/cost_and_acquisition_order.py, or re-derive every endpoint with --reuse.

The question

N12 established that the cost of denying a model an earlier position is a function of that position's residue class modulo the task's lag, and that the classes move when the lag moves. It left one thing open and said so: which residue is the expensive one is not predicted, and the ordering differs between lags.

**When each part of the task is learned, and what removing it costs.** This exercise repeats a short pattern, and the model learns the pattern one part at a time in a consistent order. Blue bars show when each part is learned; orange bars show how many extra rounds the model needs if we deny it that same part during early training. We expected the part learned last to be the expensive one to remove, on the grounds that it is the hardest. It is the other way round. The part the model learns first is the most expensive to take away, and the part it learns last is the cheapest. The likely reason is that the first part is what the rest are built on, so removing it holds up everything that follows. Worth being precise about the size: the two early parts cost clearly more than the two late ones, but the two early parts cannot be told apart from each other, so this is a split into two groups rather than a clean ranking of four.

A12 has an ordering of exactly the right shape sitting beside it. It found the model acquires the task one prefix element at a time, in a reproducible order, roughly 16 steps apart, +0.983 consistent across seeds.

Two measurements of the same object from different sides. If the expensive class is the one the model gets to last, the two orderings are the same ordering.

Kill test, fixed in advance: block cost is unrelated to acquisition order.

Design

N12 blocks a relative offset, offset k always removes the element k + 1 before the one being predicted, wherever it lands, while A12 orders absolute elements. So the intervention is re-aimed: at every scored position the model is denied the value of one named prefix element, which is what makes a cost comparable to an acquisition step for that same element. That re-aiming is why this is a new experiment rather than a re-reading of two JSON files.

Both orderings are measured in the same runs. The baseline arm supplies this configuration's own acquisition order, so the correlation never crosses run lengths, N5 established that a floor-to-ceiling midpoint is not comparable between runs of different length, and A12's runs are 900 steps against this pilot's 400.

The sign of the block is imported from N4 rather than re-derived: adding to a token's logit makes the model suppress it, because it learns to cancel the offset.

Result: related, and in the opposite direction

Six seeds, 400 steps, block applied for the first 100.

prefix element	acquired at	cost to block
0	`60.8` `[53.1, 68.6]`	`+45.8` `[+40.7, +51.0]`
1	`76.7` `[71.2, 82.1]`	`+47.5` `[+42.0, +53.0]`
2	`94.2` `[88.0, 100.3]`	`+39.2` `[+31.4, +46.9]`
3	`113.3` `[105.4, 121.2]`	`+35.8` `[+29.7, +42.0]`

The acquisition order reproduces A12's exactly: elements resolve 0 → 1 → 2 → 3, in that order, in all six seeds, about 17 steps apart.

Per-seed rank correlation between acquisition step and block cost: −0.700 [−0.815, −0.585], excluding zero. Pooled on the means: −0.800.

The kill test does not fire, cost is related to acquisition order. But the hypothesis behind it is refuted. The expensive class is the one the model acquires first, not last. The element it gets to earliest costs +45.8 steps to deny; the one it gets to last costs +35.8.

This programme has a standing rule for exactly this: an effect can satisfy a kill test while meaning the opposite of what the test was after. Both halves are reported.

What it suggests, and how firm that is

The natural reading is that the first element acquired is the one the rest are built on, so denying it is the most disruptive thing available, the model has to establish that foundation before the later elements can follow, and the block delays the whole staircase rather than one step of it. That is a hypothesis the correlation is consistent with; it is not established here.

The gradient is real but modest, and the top two are indistinguishable. Costs span +35.8 to +47.5, about 12 steps, against intervals roughly 10 steps wide. Elements 0 and 1 overlap heavily (+45.8 against +47.5); the correlation is carried by elements 2 and 3 being cheaper than 0 and 1. The honest description is a two-tier split: early pair expensive, late pair cheap, rather than a clean four-step monotone ranking, even though the rank statistic comes out at −0.700 because the per-seed orderings agree.

Verdict

The kill test does not fire: block cost tracks acquisition order at −0.700 [−0.815, −0.585].
The direction refutes the hypothesis that motivated it. Earliest-acquired is most expensive.
A12's acquisition order reproduces exactly (0 → 1 → 2 → 3, six of six seeds), in this pilot's own runs rather than across run lengths.
N12's open question has a directional answer, on one task at one lag.
The gradient is a two-tier split rather than a four-step ranking, and the record says so.

Limits

One task, one lag, one width, one learning rate, one block window and strength. N12 showed the classes move with the lag; whether the ordering of costs does is untested, and N12 already noted the ordering differs between lags, so this direction may not survive at lag 3 or 5. That is the obvious next test and it is cheap.
Four elements is a coarse rank correlation. Spearman on n = 4 takes few values, which is why the full ordering is reported beside the coefficient and the two-tier structure is stated explicitly.
Elements 0 and 1 are not separated. Any claim finer than "the early pair costs more than the late pair" is beyond what these intervals support.
The mechanism is not established. "The first element is the foundation" is consistent with the data and is not tested by it. A version that blocks an element after it has been acquired, rather than during the first 100 steps, would begin to separate "foundation" from "blocked while it was being learned".
The intervention is on the training objective, so "cost" means delay to the accuracy midpoint under this particular block, not a general measure of importance.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
baseline: The thing you compare against. A result without one is not a result.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
reproducible: Someone else running the same code on the same inputs gets the same numbers. Every result here is checked this way.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.