Your Base Model Is Better Than You Think
Most conversations about fine-tuning skip the first question. Not "how much better could this model get" but "how good is it already, on our task, before anyone touches it".
That number is cheap to obtain, and it decides whether the rest of the project is worth doing. It is also the number a vendor has the least incentive to measure carefully, because a weak baseline makes everything that follows look impressive.
We sell fine-tuning. This argues that a meaningful share of the time you should not buy it.
Download the full paper (full paper, PDF)
Untuned models are better at document work than their reputation suggests
Across six open-weight model families, before any training:
| Untuned model | Contract fields | 8-way classification |
|---|---|---|
| phi-4 | 71.9% | 79.0% |
| qwen3-8b | 69.3% | 80.0% |
| gpt-oss-20b | 68.2% | not run |
| gemma-4-e4b | 67.4% | 79.0% |
| qwen3-4b | 62.2% | 78.0% |
| granite-3.3-8b | 56.9% | 84.0% |
Random guessing on that classification task scores 12.5%. The untuned models score 78 to 84%, with no training, no examples and no project. If your requirement is "sort these into eight buckets reasonably well", an open-weight model you can download this afternoon already does most of it.
What fine-tuning is actually buying
Our measured gains, three seeds per model, every run positive: +14.1 to +24.6 points on contract extraction, and +5.3 to +11.3 on classification from an already-strong base.
The weakest starting model gains the most. On contracts the model that started at 56.9% gained 24.6 points; the one that started at 71.9% gained 14.1.
The gains are smaller where the base is already good. This is the shape you should expect. So a large advertised lift is not straightforwardly good news: it may indicate a weak starting point rather than an effective method, and the number worth comparing between vendors is the final score, not the improvement.
Four situations where fine-tuning is not worth it
1. You do not have enough examples. The same model and task on 48 training examples produced a gain of 4.5 points, inside the noise floor. On 396 examples it produced 22.3. Below a few hundred genuine examples, expect to measure nothing.
2. The base is already close to your threshold. If an untuned model does the job at 84% and your process needs 88%, you are buying 4 points. That may be worth a great deal or nothing, but it is a much smaller purchase than the pitch implies.
3. Your problem is facts, not behaviour. Fine-tuning changes how a model responds. It is poor at making it know things that change. If the requirement is "answer from our current policy documents", retrieval is the better tool: it keeps answers current without retraining and can cite the source. Measured on 200 real contracts, our retrieval reaches a 68.3% strict hit rate, 76.4% counting near-misses a human would accept.
4. You cannot say what "better" would mean. If nobody can define the held-out set and the metric, the project cannot be evaluated, and an unevaluable project will be declared a success regardless of what it does.
Measure the baseline properly, or none of this is real
Untuned numbers are remarkably easy to get wrong in a flattering direction. Three failures we have made ourselves: a model that never answered and scored zero, producing a 72.7-point phantom gain; a prompt that only the tuned model understood, producing 88 points of pure format compliance; and a model run in the wrong mode, answering with the literal wording of the instructions on 52 of 100 documents.
The practical version: read the untuned model's raw output before believing any comparison. If it did not answer, you have measured answering, not accuracy.
What we recommend instead of a default
- Measure the untuned model on your task first, on a held-out set you define, with the baseline given every reasonable chance. Days of work, not months.
- Decide what threshold the process needs before seeing what is achievable.
- If the gap is facts rather than behaviour, try retrieval first.
- If the gap is behaviour and you have a few hundred examples, fine-tuning is a reasonable bet.
- If you have neither the examples nor a definition of better, fix that first.
Step 1 is the one most often skipped, and it decides the other four.
Contract results are derived from the Contract Understanding Atticus Dataset (CUAD) v1, by Hendrycks, Burns, Chen & Ball, "CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review" (2021), The Atticus Project, used under CC BY 4.0. Complaint results use public-domain consumer complaint narratives from the US Consumer Financial Protection Bureau.