Contract AI can support extraction, comparison and triage. Whether it helps a particular practice depends on the documents, the review standard and the work needed to check its output. Test those conditions on your own documents rather than relying on a vendor demonstration.
CUAD, an expert-annotated contract-review benchmark published in 2021, asks models to identify passages for a human reviewer. It does not test autonomous legal advice or establish today's commercial product performance. Its task definition is useful here: assess whether the right evidence is found before claiming that a tool understands the contract or can decide its legal consequences.
More recent evidence is similarly bounded. The Vals Legal AI Report of February 2025 compared commercial legal AI tools with a baseline of lawyers, on US law tasks and reference answers supplied by participating law firms. On data extraction, two tools scored somewhat above the lawyer baseline; on redlining, which marks up changes to a contract, the lawyer baseline scored highest. Vendors chose which tasks to enter, and answers were graded by an automated, model-based evaluator with a second review of failing scores. The results describe those product versions and tasks: they are a reason to run your own pilot, not a substitute for one.
Choose a controlled pilot
Choose a document family that your team understands and can lawfully use in the proposed environment. NDAs can be an option, but confidentiality agreements are not inherently low risk. A missed exception, an unusual restriction or an inappropriate disclosure can matter. Use approved, representative material, including difficult documents, and confirm confidentiality, access, retention and supplier terms before uploading it.
Define the decision the pilot should inform: for example, whether reviewers can use an extraction tool to prepare a clause inventory. Keep that decision narrower than approval for autonomous contract review.
Separate assistance from legal judgment
Extraction
Potential assistance: Locate clauses and copy source passages
Required evaluation: Check omissions, references and scanned text against the original
Comparison
Potential assistance: Flag differences from an approved clause library
Required evaluation: Assess exceptions, commercial context and the governing law
Metadata
Potential assistance: Suggest parties, dates and notice periods
Required evaluation: Validate the source and relationships before updating records
Treat each output as a proposal requiring the review assigned to that task. Language support, jurisdictional coverage and document quality need testing on your own material. A general accuracy score does not establish that a tool can handle a consequential exception.
Build the clause library first
Record preferred positions, acceptable alternatives and escalation conditions. Include the jurisdiction, document type, owner and review date. Preserve links to the approved source text and record why a deviation is acceptable for a particular matter. A library needs maintenance when law, client instructions or negotiating positions change.
Test failures as well as successes
Illustrative review example
A fictional tool copies a liability clause correctly but omits a qualification in the following paragraph. The reviewer compares both passages with the source and escalates the issue. This is a test scenario, not a measured product result or a conclusion about enforceability.
Include cross-references, amendments, negations, missing pages and mixed-language material where relevant. Record false alarms, omissions, severity and the review needed to detect them. Evaluate the complete task, including preparation, checking, corrections and logging. Do not report generated output time as time saved on a completed legal review.
Set the decision gate before the trial
Define unacceptable errors, required human checks, evidence for stopping or continuing, and the person authorised to decide. There is no universal percentage that makes contract review safe. A frequent minor formatting error and a single missed material obligation need different treatment. Preserve the baseline review and unresolved issues with the decision.
Check operational fit and professional requirements
Confirm document management integration, permissions, source links, export formats and a usable review history. Ask the supplier for evidence relevant to your proposed use and verify current terms directly. Check applicable confidentiality and professional requirements, client instructions and insurance coverage with the responsible adviser or insurer. Software output does not itself establish compliance with those requirements.
Pilot review record
0/6Key Takeaway
Use a controlled pilot to establish where assistance is useful and where review remains necessary. Treat acceptance criteria as a documented professional decision, with evidence and ownership, rather than a generic accuracy target.
For help shaping a reviewable pilot, get in touch.