How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

How to Tell a Real Early Warning From a Fake One

By Joshua Bolick, Awareness Software Group ~7 min read August 2026

Here is a claim that sounds strong, and is not: every time this happened, our signal fired first.

We can say that about a measurement we spent a week on and then threw away. The same sentence is also true of a measurement we kept. Telling them apart took one extra experiment, it cost an afternoon, and it is the most useful thing in this post.

The setup

The small models we study do not improve smoothly. They sit at a low score for a long stretch, then jump to a high one quite suddenly. If you could see that jump coming, you could do useful things: stop early, allocate resources, decide whether a run is worth continuing.

So we went looking for a warning signal.

Candidate one: it looked perfect

Our first candidate measured the shape of the model's own internal numbers. It needed no test data and nothing extra to compute, which would have made it cheap enough to run constantly.

The result was about as good as results get. Eleven of the twelve quantities we measured fired before the model learned. The best one fired 74 steps early, and it did so in ten runs out of ten.

That is the point at which it is very tempting to write it up.

The test that killed it

A signal that comes before an event is not necessarily watching the event. It might just be something that happens early.

There is a simple way to tell. Move the event. If the signal is genuinely tracking, it moves too, staying roughly the same distance ahead. If it sits at a fixed point in time, it stays put while the event slides away from it.

So we changed the training settings to make learning happen much earlier or much later, shifting it across a range of 145 steps, and watched what the signal did.

Scatter plot showing the signal firing at nearly the same time regardless of when learning happened — Each dot is one training run. Along the bottom: when the model actually learned. Up the side: when the signal fired. The dashed line is what a real early warning would do. The dots do not follow it.

It barely moved. The event slid 145 steps and the signal shifted by a fraction of that, because it was really measuring the burst of change that happens at the very start of training, when an untrained model is adjusting fastest for reasons of its own.

The 74-step "lead" was just the gap between a fixed early moment and whenever learning happened to land. To use it as a warning you would already have to know the answer.

In plain English

Imagine claiming that the sunrise predicts your afternoon meetings. It is perfectly true that the sun comes up first, every single day, without exception. Move the meeting to 9am or 5pm and the sunrise does not care.

That is what our signal was doing. Reliably first, and not watching anything.

Candidate two: the same test, a different answer

The second candidate came from an established idea in physics and ecology. Systems approaching a tipping point tend to wobble more beforehand, because they recover more slowly from small disturbances as they near the edge. It shows up in climate records and in ecosystems before a collapse.

The equivalent here is how much the training signal jumps around from one step to the next. We measured it, and then we did two things before believing anything.

First we tuned the alarm against scrambled data. We took each run, shuffled its wobbles so that any build-up was destroyed while the overall shape and size stayed the same, and set the alarm's sensitivity so it went off on no more than one in five of those. Any alarm can be made to fire early by making it twitchier; this is what stops that from counting.

Then we ran the same test that killed the first candidate.

Scatter plot showing the alarm firing consistently just before learning across a wide range — The same axes as the previous figure, for the second candidate. The dots sit just below the dashed line and follow it across the whole range, which is what advance warning actually looks like.

This one tracks. It fired ahead of the event in 49 runs out of 49, across transitions that we moved by more than 200 steps, and it moved almost exactly in step with them.

It is also nearly free. The number it watches is one that training already calculates for its own purposes, so there is nothing extra to compute and no test data to hold back.

The parts we are not claiming

Two caveats, because the result is more interesting with them than without.

The warning time is not fixed. On average the alarm fires about 25 steps ahead, but that is a proportion of how long learning was going to take, not a guaranteed number. For fast runs it was as little as 2 steps. If you need a dependable amount of notice, this does not give you one.

The theory that led us here is only half confirmed. It predicts two things should happen before the jump. We could confirm one of them. Of the other two measurements, one came out flat and the other we could not test properly, because the way we set the alarm's sensitivity does not work for that kind of quantity. That is a flaw in our instrument, not evidence against the theory, and we have recorded it as untested rather than failed.

So we have a detector that works and we do not have an explanation for why. Those are different things, and the write-up says so.

The part worth stealing

The test is not clever and it is not expensive. It is one question:

If I move the thing this signal claims to predict, does the signal move with it?

It applies well beyond machine learning. Any monitoring dashboard, any leading indicator, any metric someone says "spikes before" an outage or a churn event or a failure. The obvious check is whether the signal came first. The check that matters is whether it follows.

Ours cost one afternoon and it is the only reason we did not publish a confident, well-replicated, entirely useless result.

The full record of the candidate that failed

The full record of the one that worked, with every number and the limits