How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

The Push Was a Learning Rate

By Joshua Bolick, Awareness Software Group ~11 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. Our last experiment found that pushing a small model in the direction it was already heading, just before it learned, made it learn sooner. But we had placed that push using hindsight about when learning would happen, and built it partly from information the model had not produced yet. So we removed both and compared it with simpler options.

What we found. Without the hindsight it still helped, by less. Placing it at one fixed moment for every model worked exactly as well as perfect timing. And simply making the model learn faster for those same steps, with no push at all, did exactly the same thing. Placed too early, it hurt.

Why it matters. So what began as a special direction inside the model turned out, three experiments later, to be a change in learning speed at the right stage of training. The next question is whether a faster learning speed for the whole run does just as well.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 64 training runs. The design and kill test were committed before any run.

Program v2 Bucket O, item O14. Decisive computation: analysis/momentum_push_schedule.py. Output: analysis/momentum_push_schedule.json. Reproduce with python analysis/momentum_push_schedule.py (about ten minutes on a throttled laptop CPU); --reuse re-derives every endpoint from the saved runs, and was checked to print the identical summary.

The question

O13 found that pushing a receiver along the mean of its own recent gradients for forty steps removes 19.0 steps from its transition, on all eight of J8's receivers. Its own Limits named two reasons that is not yet a speed-up: the window was placed from each run's own transition, and the averaged buffer ended at the window's close, so it included gradients from the window itself. O14 removes both and prices the push against the two plainest alternatives.

**The helpful push needs no hindsight, and a faster learning rate does the same.** The push from our last experiment, now built only from what the model had already done, placed in the 40 steps before each model learns (using hindsight), at one fixed step for every model, or at a random time. The last bar adds no push at all and simply raises the learning rate by half in the same 40 steps. One fixed step works exactly as well as perfect timing, so no hindsight is needed. And raising the learning rate does exactly what the push does: the push was a learning-rate change all along. Done too early, at a random time, it can hurt.

Every run is J8's own train(). The pushed direction is the mean of the 32 gradients before the window opens, so nothing looks ahead. Forty-step windows, alpha = 0.5:

Arm	Window
`oracle`	ending at the run's own transition
`fixed`	opening at step `112` for every receiver (the mean of the oracle openings)
`random`	opening drawn uniformly from step `40` to the oracle opening, seeded only by the receiver's seed
`lr-boost`	the oracle window, no added direction: the learning rate times `1.5`

The random windows were printed against each run's transition before any arm was trained, as the standing rule requires; all lie before the transition (openings 41-104).

Kill test, fixed before execution: the fixed arm's interval does not overlap the oracle arm's.

Anchor, in code: all eight control transitions equal J8's committed values exactly.

Result: the kill test does not fire

Change in the transition (steps; negative is sooner), J8's definition (interpolated floor-to-ceiling midpoint, 0.30 gate; every run passed it), eight receivers:

Arm	Change	Paired against `oracle`
`oracle`	`-11.75` `[-12.98, -10.51]`
`fixed`	`-11.63` `[-12.87, -10.39]`	`+0.11` `[-0.17, +0.40]`
`random`	`-2.02` `[-9.95, +5.90]`	`+9.72` `[+1.90, +17.54]`
`lr-boost`	`-11.57` `[-11.96, -11.18]`	`+0.18` `[-0.88, +1.24]`

Three findings, each one line:

Without the look-ahead the push is worth 11.75 steps, not 19.0. About 7 of O13's steps came from averaging gradients the window had not yet produced.
One fixed step does what the oracle does, to 0.11 steps -- under one evaluation interval (5 steps), so by this programme's rule the two are not distinguishable at all. The transition's spread across receivers (146-166) is small against the window, so "push just before the event" and "push around step 112" are the same instruction. K2's pattern, a fourth time.
The push is a learning-rate boost. Raising the learning rate by half in the same window, with no direction added, gives -11.57, within 0.18 steps of the push.

Timing within the run still matters: random windows average -2.02, and the three that opened earliest (steps 41, 61, 66) delayed their receivers (+13.0, +3.0, +8.8). Boosting late helps; boosting early hurts.

What stands

O13's "momentum along the model's own path" reduces, on this substrate, to "a larger learning rate in the forty steps before about step 150". The direction was never the thing; before that, the top direction was never the thing (O13).
That is a learning-rate schedule, and it is found offline: a single fixed step, set once from the control runs, matches the oracle.
What it has not been priced against is the plainest alternative of all: a higher learning rate for the whole run. J8's rate (0.002) was never tuned for time-to-transition. Generates O15.

Limits

The fixed step is set from the same eight controls it is tested on. It is one number, not fitted per receiver, but it is in-sample.
J8's substrate only, one boost size (1.5x), one window length (40).
Random windows are one draw per receiver, and their spread shows timing within the run matters; the random arm is not tight enough to say by how much.

QUALIFIED 2026-09-26 by O15. Priced against the plainest alternative, the late boost is dominated: a whole-run learning rate of 0.0025 reaches the transition 11.05 steps sooner than the boosted 0.002 run, and 0.004 reaches it 57.65 steps sooner than J8's control. The boost is a symptom of J8's untuned rate. This record's findings (fixed step matches oracle; the push is a learning-rate boost) stand. Text and numbers unchanged.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

evaluation interval: The gap between successive checks of a model during training. It is the smallest difference in timing a measurement can resolve, so two results closer together than one interval cannot be told apart.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
momentum: Keeping part of each previous training step in the next one, so training keeps moving in a consistent direction instead of zig-zagging.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.

In plain English

The question

Result: the kill test does not fire

What stands

Limits

Terms on this page

Read next

Get new results as we publish them

Want this measured on your data?