What 510 Real Contracts Taught Us About Measuring Extraction
"Our model extracts contract data with 86% accuracy" is not a claim. It is the beginning of one. Accuracy on which fields, scored how, against what reference, and what happened to the fields that are not in the number.
We fine-tuned six open-weight models on 510 real, lawyer-annotated commercial contracts. The modelling was the easy part. Deciding what could honestly be measured took longer, changed what we report, and produced a smaller number than the one we could have printed.
Download the full paper (full paper, PDF)
A contract label is a span, not a value
In this corpus each annotation is a span of the contract text that answers the field, not a normalised value. That sounds academic until you look at how long the spans are. Median reference length across 200 contracts:
| Scored | Median | Excluded | Median |
|---|---|---|---|
| Agreement Date | 16 chars | Parties | 89 chars |
| Effective Date | 18 chars | Governing Law | 146 chars |
| Document Name | 28 chars | Renewal Term | 252 chars |
| Notice Period | 317 chars |
The left column and the right column are not the same task. Pulling a date out of a header is retrieval of a short value. Reproducing a 317-character notice-period clause verbatim is something no model does reliably, and something no user actually wants.
Why we exclude the interesting-sounding fields
The obvious metric is exact match: did the model produce the reference string. On a date that is exactly right. On a clause it is worse than useless. A model can locate the notice-period clause perfectly, summarise it accurately, and start its answer three words earlier than the annotator did. Exact match scores that zero.
So we score the short value-like fields and exclude the long clause spans, and we state the exclusion in the same sentence as the number:
On 510 public lawyer-annotated contracts, fine-tuning improved short-field extraction from 71.9% to 86.0%. Clause-span location is a different task and is not measured.
Not "86% accurate on contract understanding".
This makes our number smaller. A vendor scoring everything with exact match would publish a lower headline; one scoring clauses with a generous overlap metric could publish a higher one. Both would report something other than what a buyer will experience. The rule we hold to: if a metric cannot fairly score something, exclude it and say so in the same breath. A buyer who discovers the exclusion themselves will reasonably discount everything else you have told them.
Two defects the real corpus found and our tests did not
We built the data pipeline against a hand-written fixture. Every test passed. Two real defects were sitting in it, and both appeared only when it ran against all 510 contracts.
Multi-answer fields were truncated to the first span. 508 of the 510 contracts label Parties with several spans, because a contract usually has more than one party. The code took the first. The reference for a two-party agreement became one name, and any model that correctly answered with both was marked wrong.
About half of all answer spans sat past the point where we truncate. Contracts are long and context windows are finite, so the pipeline truncates. But it was still scoring the model against answers that lived in the truncated-away part of the document. That is not a hard metric. It is a measurement of our own truncation, dressed as a measurement of the model.
The fix keeps per-span offsets so a reference can be rebuilt from only the spans the truncated context actually contains. With the window we use, roughly 84% of scorable references survive that filter. We report that figure rather than the more flattering one we first published, which was wrong.
Fixtures prove shape, not correctness. A test suite built on hand-written examples confirms the code does what its author expected. It cannot tell you what the real corpus contains. Both defects were invisible to a green test run and obvious within minutes of looking at real output.
Questions for any extraction vendor
- Which fields are in the number, and which are not? Ask for the excluded list. If there is not one, ask how long clause fields are being scored.
- What is the reference? A normalised value and a span of source text are different targets, and only one can be exact-matched fairly.
- Was the model shown the text containing the answer? Truncation is universal for long documents.
- Is a multi-part answer scored as one thing? Parties, jurisdictions and notice periods are frequently plural.
- What does the metric do with a nearly-right answer? Exact match says zero. Whether that is correct depends on your workflow.
A vendor who has thought about their metric will have answers and will volunteer the limits.
Contract results are derived from the Contract Understanding Atticus Dataset (CUAD) v1, by Hendrycks, Burns, Chen & Ball, "CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review" (2021), The Atticus Project, used under CC BY 4.0. Complaint results use public-domain consumer complaint narratives from the US Consumer Financial Protection Bureau.