How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

The Archive Was Too Coarse

By Joshua Bolick, Awareness Software Group ~14 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: How much model does a task need, and what changes when it has more? – The abrupt jump is what spare capacity buys -- it fades smoothly as the model shrinks, long before the model stops working. Capacity and task difficulty act separately, not as a ratio.

In plain English

What we asked. Everything we had found so far came from tiny models on made-up tasks. So we took two small public language models whose makers saved copies all through training, and tested every saved copy on simple copying tasks, without training anything ourselves.

What we found. Both models learned to copy somewhere between two consecutive saved copies, with nothing saved in between. So the archive cannot tell us which skill came first or how sudden the change was. A third test was settled by its own rule before it ran, which we caught and have now set up properly. One hint survived: the model's internal state reorganises just after the skill appears.

Why it matters. The thing worth copying: before using someone else's saved snapshots to time an event, check whether the event falls between two of them. Here it did, in both models.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. No training: 48 published checkpoints of two public models, measured on a laptop CPU, each downloaded, measured and deleted (about 12 GB streamed). The design, the three endpoints and the kill test were committed (8fa5d25) before any checkpoint was downloaded.

Program v2 Bucket O, item O1, option A (the smallest pass, chosen by the maintainer on 2026-09-26). Decisive computation: analysis/pythia_archive.py. Output: analysis/pythia_archive.json. Post-hoc diagnosis, written after the run and changing no verdict: analysis/pythia_archive_posthoc.py -> analysis/pythia_archive_posthoc.json. --reuse re-derives every endpoint from the saved measurements and was checked to print the identical verdict.

The question

Every result in this programme is on tiny recurrent networks and synthetic tasks. O1 asks whether three things this programme found appear in someone else's real language models: EleutherAI's Pythia-70M and Pythia-160M (deduplicated), which publish checkpoints through training on identical data in identical order.

**Both models learn to copy between two published snapshots.** Two small public language models were saved many times during training. We tested each saved copy on two copying tasks using random words it could not have memorised. Solid lines: copying a whole repeated sequence. Dashed: continuing one repeated word. The vertical lines mark two consecutive saved copies. Both skills appear in both models between step 512 and step 1,000, and nothing was saved in between. The archive cannot say which skill came first or how sudden the jump was. The smaller model also later loses much of its copying ability.

**The spread of the internal state starts highest, collapses, then peaks again.** How many independent directions the model's last internal layer spreads its information across, measured on the same copying test. A freshly initialised model spreads almost evenly, so it scores highest of all at the start. Our rule looked for the highest point anywhere, and the untrained start always wins that, so the test was settled before it ran. The pattern worth checking is the second peak, one to two snapshots after the skill appears. That is now preregistered on independent training runs.

Two probes on random tokens, so nothing can be recalled from training data: repeat (one token repeated; the model should continue the run) and induction (a random sequence of 32 tokens, then the same sequence again; predicting the second copy needs the model to look back and copy). The three phenomena, and the rule for each, fixed before execution:

Acquisition order (D2, P3, O11): the simpler skill is acquired at an earlier checkpoint than the harder one.
Sharpness against width (N6): the larger model's induction transition, 10% to 90% of its own peak in steps, is narrower.
A spread peak at the transition (N9): either spread measure of the final hidden states (effective dimension, spectral entropy) reaches its maximum at an interior checkpoint within one of induction's acquisition.

Kill test: none of the three appears in either model. Precondition: induction peaks at 0.5 or more in both models. It passes (0.826 and 0.923).

Result: the kill test fires

	Pythia-70M	Pythia-160M
Induction peak (step)	`0.826` (`13000`)	`0.923` (`33000`)
Induction at the last checkpoint	`0.483`	`0.874`
Induction acquired (half its peak)	step `1000`	step `1000`
Repeat acquired (half its peak)	step `1000`	step `1000`
Induction `10%`-`90%` width	`4000` steps	`5000` steps
Effective dimension: global maximum	step `32`	step `0`
Spectral entropy: global maximum	step `0`	step `0`

None of the three appears by its rule. But none of the three was decided by the models. The post-hoc diagnosis below shows what decided each one, and it is the part of this record worth keeping.

What decided each verdict

Order and sharpness: the archive's grid. Pythia publishes steps 0, 1, 2, 4, ... 512, then every 1000. In both models, both probes go from nothing at step 512 to most of their peak at step 1000: induction from 0.000 to 0.546 and 0.612. Everything this programme would call the transition happens inside one gap in the archive. Two skills acquired inside the same gap cannot be ordered (Q7's rule: inside one evaluation interval is undecided, not a tie). The 10%-90% width therefore measures not the jump but the slow climb after it, which is a different thing.

The spread peak: the rule's use of the global maximum. At initialisation a random network's representations are spread almost evenly across dimensions, so a global maximum sits at or near step 0 whatever training does: 154.9 and 192.6 effective dimensions at step 0. That decided the verdict before any data arrived -- another case of a design deciding its own answer, and a rule carried over from GRUs, whose spread starts low. What the curves actually do is collapse over steps 64-256 (to 15.0 and 16.6 effective dimensions) and rise again through the transition, peaking one or two checkpoints after it (step 2000 for Pythia-70M, 3000 for Pythia-160M) before falling to almost nothing by the end (8.7 and 2.2). That post-collapse peak is post-hoc and one seed per size; O25 preregisters it on independent seeds.

Two things measured on the way

Pythia-70M forgets how to copy. Its induction accuracy peaks at 0.826 near step 13000 and falls to 0.483 by the end; Pythia-160M holds 0.87-0.92. Continuing a repeated token is lost by both after step 3000. A capability measured only at the last checkpoint would understate what these models learned.
Both models' final hidden states collapse onto a few directions by the last checkpoint. That is consistent with the known anisotropy of trained language-model representations, and it means any representation measure taken only at the end reads a different object from one taken during training.

What stands

Kill test fires, and it does not show these phenomena are absent from Pythia. The archive's step-512 to step-1000 gap contains both models' whole transition, so order and sharpness are undecided at this grid; the spread rule was decided by initialisation.
The post-collapse spread peak, one to two checkpoints after the transition in both sizes, is the one pattern that survives, as a post-hoc description.
Practical reading: a public checkpoint archive answers "did it learn" well and "when, and in what order" only as finely as its checkpoint spacing. Look at where the event falls on the grid before designing against one.

Limits

Two sizes, one seed each, one pair of probes. PolyPythias seeds exist (O25 uses them).
The early transition cannot be resolved from the public archive at all. Finer timing would need a re-trained small Pythia with dense early checkpoints (arXiv 2510.12071 did this for Pythia-14M), which is a GPU item and the maintainer's decision (O26).

QUALIFIED 2026-09-27 by O25. On five independent Pythia-70M seeds the post-collapse maximum fell one or two checkpoints after the transition on three, not the four the kill test needed. On all five the spread has reached at least 97% of its highest value by step 2000 and stays near it: the "peak" above was the first point of a plateau. Text and numbers unchanged.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
checkpoint: A saved copy of a model partway through training, so you can come back and inspect what it looked like at that moment.
effective dimension: Roughly, how many independent directions a model's internal state actually uses. High means spread out; low means squeezed onto a few.
evaluation interval: The gap between successive checks of a model during training. It is the smallest difference in timing a measurement can resolve, so two results closer together than one interval cannot be told apart.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
hidden state: The model's working memory: the internal numbers it carries from one step of a sequence to the next.
induction: A model's ability to spot that a sequence is repeating and copy what came next last time. A standard early skill in language models.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
plateau: A stretch of training where the model is not visibly improving. Often, though not always, followed by a sudden jump.
post hoc: Worked out after the fact, rather than decided in advance. We report such checks separately and never let them decide a result, because it is far too easy to find a pattern once you already know the answer.
probe: A small separate model trained to read information out of a bigger model's internals, used as a measuring instrument rather than as a product.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
sharpness: How abrupt a model's improvement is. We measure the biggest change over any short stretch of training as a share of the run's total change, scaled so that a model improving at a perfectly steady rate would score 1. A score of 20 means the fastest stretch was twenty times steeper than a straight line.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.