How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

The Archive as an Instrument

By Joshua Bolick, Awareness Software Group ~11 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

EXPLORATORY. Not a preregistered study. No new experiments, no training, no GPU, no cost. Reads only the committed results/ bundles.

Program v2 tier 0. Three backlog items in one pass, because they share one loader:

F10, build the meta-analysis harness over the archive.
A9, does when the transition happens predict what the run ends up knowing?
E4: first fit of a transition scaling law, and its variance decomposition.

Decisive computation: analysis/archive_instrument.py over src/latent_pretraining/analysis/archive.py. Output: analysis/archive_instrument.json. Reproduce with python analysis/archive_instrument.py in about ten seconds.

F10: the instrument

latent_pretraining.analysis.archive is now the single loader for committed bundles. It reads experiment.json, run.json, summary.json, metrics.csv, and analysis/representation_trajectory.json into typed records, derives the preregistered transition features once, and emits three tidy tables: per run, per logged step, per capture step and layer. It is standard-library only and does not import torch, so the archive stays readable in an environment that cannot train.

**What a learning transition looks like.** Two things measured at every step of a single training run. The blue line is how often the model gets the answer right on data it has never seen. The orange line is 'residual energy', how much of the model's internal activity lies *outside* the eight directions it uses most. Low residual energy means the model's internal state is compact and concentrated; high means it is spread out. The model sits at chance for about 80 steps, then learns the task in a burst of roughly 40 steps. That burst is what this project calls the transition. Learning here is not gradual. It is a sudden jump, and the model's internal state visibly reorganises at the same moment, the orange line spikes right as the blue line climbs. Everything else in this program is an attempt to work out what that spike is and whether it can be used.

**A strong correlation that turned out to mean nothing.** Each dot is one training run: when it learned the task (left to right) against how accurate it ended up (bottom to top). Looking at all the dots together, there is a strong downward trend, runs that learned later ended up worse. But the colours show why that is misleading: model width drives both. Wider models learn earlier *and* end up more accurate, so the trend across all dots is really just the width effect in disguise. Within any single colour the trend collapses, and the three colours even point in different directions. The apparent finding was an artifact of mixing three different model sizes together. This is why the result was recorded as 'not answerable from the archive' rather than as a discovery.

**When the transition happens is set by model size, not by luck.** Each bar is the average step at which models of a given size suddenly learn the task. The whiskers show how much that varies between runs that differ only in their random starting point. Wider models learn earlier: around step 111 at width 36, step 72 at width 72. The gap between the bars (about 39 steps) is roughly seven times larger than the whiskers (about 5 steps). So the moment of learning is mostly a consequence of how the model was configured, not of chance, which means it can be predicted in advance from the settings alone, to within about ten steps.

**What 183 committed experiments actually cover.** Every experiment this project has run and kept, arranged by which study it belonged to and how wide the model was. Darker cells hold more runs. Blank cells were never run. The archive is deep but narrow. Model width is the only setting that was ever varied: learning rate, task difficulty, and vocabulary size are identical in all 133 runs of the main task. That is why a planned analysis of how timing depends on those settings could only be done for width, and why future sweeps should be saved as full experiments rather than throwaway scripts.

**The internal signal arrives with the transition, never before it.** For every archived run we found two moments: when accuracy jumped, and when the internal 'residual energy' peaked. This counts the gap between them. Bars to the right of zero mean the internal signal peaked *after* the model had already started getting answers right; bars to the left would mean it peaked first. Almost everything sits at or after zero. The internal signal is a companion to learning, not a warning of it. That is the single biggest obstacle in this research program: you cannot use a signal to act *before* an event if the signal only shows up once the event is underway.

This existed to be built because every pilot in analysis/ carried its own bundle walk, its own transition definition, and its own t-critical table. Four of the five discrepancies the 2026-08-22 review found were consequences of that duplication. There is now one definition of each, unit-tested in tests/unit/test_archive.py.

Regression check. Re-deriving the published H6 screen through the harness reproduces it exactly:

Quantity	Harness	Published (H6 pilot)
Runs	41	41
Rise above own baseline	`+0.2131` `[+0.2082, +0.2181]`	`+0.2131` `[+0.2082, +0.2181]`
Peak offset from transition	`+14.1` `[+8.9, +19.4]`	`+14.1` `[+8.9, +19.4]`
Peak within 50 steps	100%	100%

Inventory. 183 bundles: 133 tiny-gru on synthetic-delayed-copy-v1, 50 tiny-transformer on synthetic-v1. 133 logged held-out accuracy, 175 captured a representation trajectory, 57 are task-only, and 41 are task-only and eligible for transition work.

One thing the harness settles immediately, and it is not good news

The backlog's E4 entry assumed "the archive already seeds three of the four axes free". It does not. Across every committed delayed-copy bundle, only three quantities were ever varied: latent width (36 / 48 / 72), penalty weight (0 / 0.025 / 0.05), and retained rank (tied to width at W/6). Learning rate is 0.005, lag is 4, vocabulary is 32, and batch size is 64 in all 133 runs. The lr, lag, and vocabulary sweeps this project has published came from CPU pilots that wrote JSON, not bundles, so their trajectories are not in the archive at all.

That is a standing cost of the pilot convention and it is worth stating plainly: cheap pilots buy answers but do not accumulate an archive. Anything wanting to re-analyse a pilot later must re-run it.

A9: does transition timing matter to the outcome?

Kill test as written: |Spearman rho| < 0.3 between transition step and final held-out accuracy across eligible archived task-only runs.

Verdict: not answerable from the archive. Not a null and not a signal, the measurement does not have the resolution the question needs, in two independent ways.

The pooled correlation across all 41 runs is -0.679, which looks decisive and is worthless: width sets both variables. Wider models transition earlier (72.5 steps at width 72, 111.2 at width 36) and finish more accurate. Stratifying by width is mandatory, and it dissolves the result:

Width	Runs	rho	Distinct transition values	Modal tie	Final-accuracy spread
36	8	`-0.316`	3	62%	`0.0485`
48	25	`+0.503`	3	72%	`0.0188` (ceiling-limited)
72	8	`-0.444`	2	75%	`0.0048` (ceiling-limited)

The signs disagree. Only width 48 clears the 0.3 threshold, and it does so in the opposite direction to both other widths. Within width 48 the three contributing studies agree in sign (+0.504, +0.866, +0.308) but each is built on the same two problems:

Timing is quantised to the logging interval. Every archived study logged every 10 steps, so the transition step takes 2–3 distinct values within a width and 72–75% of runs share the modal value. A rank correlation over three tied levels is a two- or three-group mean comparison wearing a correlation's clothes.
The outcome is at the ceiling. Final accuracy spans 0.0048 at width 72 and 0.0188 at width 48. There is almost nothing for timing to rank. This is the saturation problem CLAUDE.md already warns about, showing up in a new place.

Zero of the three strata have both enough timing levels and enough outcome spread to support the test. Reporting -0.679 as an A9 answer would have been this project's third confounded headline.

Second question: does anything at initialisation predict transition time? Pooled, this looks even stronger (initial final_norm effective rank against transition step: rho = -0.776). It is the same width confound: wider models start with higher effective rank and transition earlier. Within width, everything collapses:

Width	Strongest initialisation predictor	rho	n
36	`initial_loss`	`+0.591`	8
48	`init_top_energy_share[embedding]`	`-0.511`	25
72	`initial_loss`	`+0.252`	8

No predictor is consistent across widths, and the two largest come from n = 8 strata where the 5% critical value for Spearman is about 0.71. Consistent with K1: initialisation does not predict transition time either.

What A9 actually needs, and it is cheap: a dedicated pilot at log_every = 1, on a task with real headroom (the E1 battery), across enough seeds to resolve a correlation. That is CPU minutes. A9 moves from "free, archive-only" to "cheap, needs tier 1": it depends on E1.

E4: first fit of a transition scaling law

Kill test as written: across-seed variance at fixed hyperparameters swamps the hyperparameter dependence, making prediction useless at the precision Bucket C needs.

Verdict: survives, on the one axis the archive varies.

Width	Runs	Mean transition step	SD	Range
36	8	111.2	6.4	100–120
48	25	92.0	5.0	80–100
72	8	72.5	4.6	70–80

Width explains eta^2 = 0.853 of transition-step variance.
Between-cell mean spread: 38.8 steps. Within-cell spread: 5.2 steps, falling to 4.4 once the 10-step logging quantisation is removed in quadrature. Ratio 7.4x.
Power law: T ~ width^-0.615, R^2 = 0.859.
Implied scheduling precision: ±10 steps at 95%.

The exponent is the least trustworthy number here, three width points cannot establish a functional form, and the fit is reported for continuity with future sweeps, not as a law. The variance decomposition is the load-bearing part, and it says something the program cares about a great deal.

Program v2's central obstacle is that no online signal leads the transition (the geometry peak sits +2.1 steps after it). E4's premise was that this may not matter if timing is predictable from hyperparameters alone, and set the bar at "within a few hundred steps". On the width axis, at fixed everything else, it is predictable to ±10 steps, an order of magnitude better than the bar, on a transition that lands around step 90.

What this does and does not license. It does not deliver a leading indicator: predicting from hyperparameters is an offline schedule, not an online signal, and it is fitted on one axis with three points, at one architecture, on one task, at one learning rate. It does mean the oracle-timing version of A4 (the transition kick) has a non-oracle successor that is not obviously dead, and that a random-time-matched control is testable against a scheduled arm rather than only a retrospective one. The next thing E4 needs is the lr / lag / vocabulary axes measured as committed bundles rather than uncommitted pilots.

What changed in the backlog

Item	Before	After
F10	tier 0, platform	done
A9	tier 0, free, archive-only	archive cannot answer it; needs E1 headroom task + `log_every=1`; moves behind tier 1
E4	tier 3-ish consequence, gated	first fit survives; width dependence is 7.4x seed noise, ±10-step precision; needs the other three axes as bundles
E1	tier 1 platform	now also gates A9

Standing caveat carried forward: everything above is one architecture (tiny-gru), one task (synthetic-delayed-copy-v1), one optimizer (AdamW), one learning rate. D8 (does the phenomenon survive SGD?) and E3 (the architecture × task grid) still scope all of it.

The plain-language version. This page is the record: the design, the numbers and the limits. If you would rather have the story, The Archive as an Instrument, explained without the jargon, covers the same result for a reader who has never seen a training curve.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

All research from this programme →

F10: the instrument

One thing the harness settles immediately, and it is not good news

A9: does transition timing matter to the outcome?

E4: first fit of a transition scaling law

What changed in the backlog

Want this measured on your data?