How We Prove Your Model Is Actually Better (Not Just Different)
Ask most AI vendors how they know their model is better, and you'll get a demo and a good feeling. Ask for a number, and the room goes quiet.
The AI industry runs on impressive-sounding claims. A model is "state of the art," it "tops the benchmarks," it "feels smarter." None of that tells you whether it will do your job better than what you have today. If you're going to bet a workflow, or a compliance posture, on a model, you deserve evidence, not vibes.
Here's the method we hold ourselves to, and a worked example with real numbers, including the two results we had to throw away.
Why public benchmarks don't answer your question
Public AI benchmarks measure general ability on standardised tests: trivia, math puzzles, coding challenges. They're useful for comparing models in the abstract, but they have two problems for a real buyer.
They're not your work. A model that aces a general benchmark can still be mediocre at reading your contracts, answering your support tickets, or following your policies. The benchmark measures a different job than the one you're hiring the model to do.
They can be gamed. Because benchmarks are public, models increasingly get tuned to look good on them specifically. A high score can reflect test-prep as much as genuine capability.
The only benchmark that answers your question is built from your tasks. Everything else is a proxy.
The method: measure tuned vs. base on held-out data
It's simple, and its power is entirely in the discipline of actually doing it.
1. Build the test from real work. Before training, set aside a portion of the real examples (actual questions with the answers your experts would give) as a held-out set. Held-out means the model never sees them during training. It's the equivalent of testing a student on questions that weren't on the study guide.
2. Score the original model first. Run the plain, un-tuned base model on those held-out questions and record how it does. This is the honest starting line, and skipping it is how most "AI improved our process" claims quietly become unfalsifiable.
3. Fine-tune, then score again, using the same questions and the same scoring.
4. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it. Not a vibe: a measured delta.
A worked example, with the actual numbers
Here is that method applied to a task we can share publicly, so you can check the work rather than take our word for it.
The dataset is CUAD: 510 real commercial contracts, annotated under the supervision of practising lawyers, published openly by The Atticus Project. The task: pull specific fields out of a contract (document name, agreement date, effective date). The model: Qwen3-4B, a small open-weight model that runs comfortably on hardware a mid-size firm already owns.
396 contracts for training, 99 held out, split so no contract appears on both sides. We ran it three times with different random seeds, because a single run tells you nothing about whether a result is stable.
| Run | Base model | Fine-tuned | Lift |
|---|---|---|---|
| 1 | 62.2% | 85.4% | +23.2 |
| 2 | 62.2% | 84.6% | +22.5 |
| 3 | 62.2% | 83.5% | +21.3 |
A base model does this job at 62%. The same model, fine-tuned on a few hundred examples, does it at 83–85%, an average gain of 22 points, reproducing across three runs to within two points.
Why the base score is identical every time: the held-out questions and the un-tuned model don't change between runs, so the starting line shouldn't move. If it had, something would be leaking between the training and test sets, and the result would be worthless. Checking the boring numbers is how you catch that.
The part most vendors won't show you: the results we threw away
Before that 22-point result, this same pipeline produced two numbers that looked better and were completely fake. We caught both. Here's how, because the failure mode is common and you should know how to test for it.
A 72-point lift that wasn't real. An early run reported the base model scoring zero and the tuned model scoring 72.7%. Spectacular, and nonsense. The base model was a reasoning model that spent its entire response budget "thinking out loud" and never got around to emitting an actual answer in the required format. It was scored zero for a formatting failure, not an inability to do the task. Its own reasoning text contained the right answers.
A 45-point lift that reversed. On a different dataset, a fine-tune appeared to add 45 points. Once we gave the base model a fair chance to answer, the same comparison came out at minus 9. Fine-tuning had made it worse, and we'd nearly reported the opposite.
The tell in both cases was the same, and it's a question you can ask any vendor: did the baseline actually answer? For the 22-point result above, we checked every one of the 99 held-out questions. The base model produced a valid, well-formed answer to all 99, at essentially the same length as the tuned model's. It was beaten fairly.
The question to ask a vendor: "Show me what the un-tuned model actually produced on your test set." If the baseline was failing to respond at all, the lift is measuring formatting, not capability, and the number is meaningless.
How much of your data do you actually need?
This is the question we get most often, and the measurements answer it more usefully than the headline does.
The same model, the same task, trained on 48 examples instead of 396, produced a gain small enough to be indistinguishable from noise. At 396 examples it produced 22 points. The base model's score never moved.
That's the finding worth carrying: for this kind of task, the constraint wasn't the model's ceiling. It was having enough examples. A few hundred good ones changed the outcome completely. That's a far smaller number than most organisations assume, and it's usually sitting in a folder somewhere already.
What this does and doesn't tell you
Stating the limits is part of the method. A number without its caveats is marketing.
This measures short-field extraction: document name, agreement date, effective date. Locating a long clause span (a full governing-law or notice-period paragraph) is a genuinely different task needing a different metric, and we don't claim a number for it.
These are public contracts, not yours. CUAD is drawn from public filings. Your documents have different formats, house style, and vocabulary. What this demonstrates is that the method works and roughly how much data it takes, not what your accuracy will be.
Update: we finished the rest, and published it. This article originally said "one model, one domain, so far" and promised the others would be published the same way. They were. Six model families, three task types, three random seeds each: 48 measured runs, every one an improvement. That write-up also includes the two claims we had to withdraw and six results of our own that turned out to be wrong. Read the full results.
That last point matters more than it looks. A results table with no failures in it isn't a measurement. It's a brochure. We record in advance which experiments we expect to fail, so a null result is evidence rather than an embarrassment.
The bottom line
Don't accept "it's better" on faith. A credible AI engagement proves improvement the way any serious discipline does: define the test from real work, measure the baseline, make the change, measure again, and report the honest difference, caveats attached.
If a vendor can't show you that number, on your tasks, on data the model never trained on, you don't have evidence. You have a demo.
And if they show you a number without showing you what the baseline produced, ask.
Contract results above are derived from the Contract Understanding Atticus Dataset (CUAD) v1, by Hendrycks, Burns, Chen & Ball, "CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review" (2021), The Atticus Project, used under CC BY 4.0.