How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Measured results

What 48 Fine-Tuning Experiments Showed

By Joshua Bolick, Awareness Software Group ~8 min read August 2026

Most published claims about fine-tuning are a single number from a single run. That is not enough to plan with, because it cannot tell you whether the result was the method or the luck.

We ran 48 controlled experiments to find out what fine-tuning actually does to an open-weight model on document work. Six model families, three random seeds each, two publishable task types, every run measured against the same untuned model on the same held-out data.

Download the full paper (full paper, PDF)

How the measurements were made

Each experiment fine-tunes one open-weight base model on one corpus, then scores the tuned model and the untuned base on the same held-out records, with the same prompts and the same metric. Records are split by source document, so no document appears in both training and evaluation.

Three details matter more than they sound:

The baseline is run twice and the better score is kept. Several of these models can run in a step-by-step reasoning mode or a direct mode, and which one is stronger changes by task. Picking one in advance would let us choose a weak baseline and report the difference as our contribution.

Three seeds per cell, not one. Random seed changes the result. On one model the same configuration on the same data moved 6.0 points between seeds. A single run cannot tell you whether a 3-point gain is real.

Every number is recomputed from the raw model outputs, not read from the metrics the evaluation wrote.

Result 1: every run improved

All 48 cells showed positive lift. That is the least interesting finding and the one worth stating first, because it sets the floor.

Contract field extraction

510 real, lawyer-annotated commercial contracts. Scoring short value-like fields: document name, agreement date, effective date. 99 held-out contracts, three seeds.

Model	Untuned	Fine-tuned	Gain
phi-4 (14B)	71.9%	86.0%	+14.1
qwen3-8b (8B)	69.3%	85.9%	+16.6
gemma-4-e4b (4B)	67.4%	84.8%	+17.4
gpt-oss-20b (20B)	68.2%	84.6%	+16.5
qwen3-4b (4B)	62.2%	84.5%	+22.3
granite-3.3-8b (8B)	56.9%	81.5%	+24.6

Complaint classification

100 held-out consumer complaint narratives, each assigned to one of eight product categories. A different shape of task: the answer is a judgement about the whole document and appears nowhere in its text.

Model	Untuned	Fine-tuned	Gain
granite-3.3-8b	84.0%	89.3%	+5.3
qwen3-4b	78.0%	89.3%	+11.3
gemma-4-e4b	79.0%	88.0%	+9.0
phi-4	79.0%	87.7%	+8.7
qwen3-8b	80.0%	85.7%	+5.7

Random guessing scores 12.5% here and the untuned models score 78 to 84%, so this is a much harder baseline to beat than the contract task. Gains against a model that is already good are worth more than gains against one that cannot answer.

We also measured a third task type, key-value extraction from scanned forms. Those numbers are not published because the corpus is licensed for research use only. They are internal evidence, and treating them otherwise would be a licence violation regardless of how convenient the result is.

Result 2: fine-tuning compresses the differences between models

Contracts	Untuned	Fine-tuned
Best model	71.9%	86.0%
Worst model	56.9%	81.5%
Spread	15.0 points	4.5 points

The weakest starting model gained the most, 24.6 points, and finished within 5 points of the best one. Which base model you start from matters considerably less than whether it is trained on your data. That is useful, because base model choice is the part of this decision a buyer is least equipped to evaluate.

The counter-example, stated in the same breath. This does not hold everywhere. On the third task we measured, the spread between models widened after training instead of narrowing. We are not publishing that task's figures, for the licence reason above, which means you are being shown the result that supports our point and told without numbers about the one that does not. Weigh it accordingly.

Result 3: two things we got wrong

An evidence table with no failures in it is a marketing document.

We predicted every forms cell would fail. All of them improved. The reasoning was that the task names the target fields in the prompt, so there is no schema to learn, and training would help formatting at most. The prediction was wrong by 10 to 20 points per cell. What we had underestimated is that teaching a model how to answer is worth a great deal on its own.

We published a pattern that held on one task and broke on another. We had claimed the ranking by gain was the exact reverse of the ranking by final score, five models out of five. It holds on contracts. It fails on the next task. The claim was over-generalised and has been withdrawn.

What these numbers do not cover

Short fields, not clauses. The contract results score short value-like fields. Long clause spans are excluded, because exact match scores a 300-character clause near zero no matter how well the model located it. The supportable claim is "short field extraction at 86.0%", not "contract understanding".

Public benchmarks, not your documents. These are public SEC-filed contracts. Your agreements have different house style, formats and clause vocabulary. These numbers say the method works and roughly how much it is worth. They do not predict a result on a corpus we have not seen.

One training recipe. Every number comes from a single configuration, unchanged across all 48 runs, which is what makes the models comparable. Nobody has tuned it, and the training curves suggest these are a floor rather than a ceiling.

What we take from it

On document work with checkable answers, fine-tuning an open-weight model on the target data reliably helps, and the size of the help is 10 to 25 points depending on where the model starts. A model that starts weakest often gains most, which means an unimpressive off-the-shelf demo is weak evidence about the trained result. And any single reported lift should be treated as unverified until you know what the baseline was allowed to do, and whether it answered at all.

Contract results are derived from the Contract Understanding Atticus Dataset (CUAD) v1, by Hendrycks, Burns, Chen & Ball, "CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review" (2021), The Atticus Project, used under CC BY 4.0. Complaint results use public-domain consumer complaint narratives from the US Consumer Financial Protection Bureau.