ContractBench tests whether AI makes the calls an experienced contracts lawyer would. The best score was 58.3%. The gaps matter more than the ranking.
Can AI mark up a contract like an experienced lawyer? Not yet. On ContractBench, a benchmark contract AI company Ivo and data lab micro1 published on October 13, 2026, all seven models tested scored below 60% on weighted attorney-defined criteria, with the best at 58.3% and the top three within 1.2 points, according to Ivo’s research report. Artificial Lawyer covered the results on October 23, 2026.
How Did Ivo Test Contract Review?
ContractBench has 350 tasks across 35 scenarios and five agreement types: master services agreements, IP licenses, research partnerships, commercial leases and marketing agreements. Thirty scenarios use U.S. law and five use UK law. Half the tasks are a first pass on a clean draft. The other half ask the model to respond to a counterparty’s markup.
Attorneys wrote the rubrics after doing each redline themselves. Every rubric item asks whether the reviewer took the right action on a clause: edit it, leave it alone, comment, or escalate it to someone with authority. High-priority items carry the most weight. For first-pass tasks, five attorneys independently reviewed the same contract, so models were scored against five reasonable answers instead of one.
Keep one design choice in mind as you read the results. The rubrics are attorney-written, but the pass or fail grading was done by three AI judges from different providers, with a majority deciding. Ivo reports the judges agreed unanimously on 83.4% of items.
Where Did the Models Fall Short?
The top three were close. Grok 4.7 scored 58.3%, Opus 5.5 58.1% and Kimi K3 57.1%, per the micro1 results page. Gemini 3.8 Flash and GLM-5.3 followed at about 55%, then DeepSeek V4 Pro at 48.4% and GPT-6 Astra at 45.6%. Completion rates varied: Opus 5.5 completed 80.8% of its runs, with most failures caused by hitting the output limit, while Grok finished 99.5% and Kimi all of them. Ivo says Kimi cost 38% less per completed run than Grok.
Honestly, the ranking is the least interesting part. On counterparty markups, every model passed more than 96% of items where the right move was to leave untouched text untouched, but on items that called for a new edit, scores fell to between 15.1% and 36.5%. Six of seven models scored lowest on the issues attorneys rated most important.
Context was hard too. Every model scored 9 to 15 points lower when a playbook rule had to be applied to the specific facts of the deal than when the rule applied no matter what, and six of seven did worst with no playbook at all. One surprise: GPT-6 Astra ranked last overall but passed 70.8% of escalation items, more than twice any other model, making it the best at recognizing when a call was above its authority.
And the models don’t redline like lawyers. Attorneys made 63.5% of their edits inline, while the best model managed 36.6%. On first-pass reviews, attorneys averaged 125 edits per task at 93 characters each; models averaged 41 edits at 300 characters. Any lawyer who has received a “markup” that’s really a rewrite will recognize the problem.
How Much Weight Should Buyers Give a Vendor’s Benchmark?
We’d treat it as a strong signal with two caveats. First, Ivo sells contract review AI, so it has an interest in showing that raw models fall short. The benchmark tests base models through one shared test setup; it does not test Ivo’s product or any other vendor’s. Second, AI judges did the grading. The high agreement rate helps, but attorneys did not score every answer.
Even so, the weak spots are specific enough to test on your own contracts: high-priority clauses, deal-specific judgment, counterparty markups, and edits that rewrite a clause when a word would do.
What Should Your Team Do With These Results?
If you run a large in-house team using AI for first-pass review, spot-check the high-priority clauses hardest, because that is where these models scored lowest, and ask vendors which model sits underneath and whether they have tested it against something like this. Haven’t written down your positions and fallbacks yet? Playbooks helped nearly every model, so that work comes before buying a tool that depends on them. Small teams and solo counsel should be most careful with counterparty markups, where every model’s scores dropped. And if you were hoping for a single winner to standardize on, the top three sit within 1.2 points of each other.
We rate ContractBench Real: the method is careful and the gaps it exposes are the ones buyers should test for. Discount the leaderboard, and remember that the best model met fewer than 60% of weighted attorney criteria.
FAQ
What is ContractBench?
A benchmark from Ivo and micro1, published October 13, 2026, testing whether AI models make the same judgment calls as experienced attorneys reviewing and negotiating contracts, per Ivo’s report.
What was the top score?
58.3%, on weighted attorney-defined criteria, per the October 13, 2026 report.
Did playbooks help?
Yes. Playbooks helped nearly every model in the test.
Where did models struggle most?
On high-priority clauses and on counterparty markups, where every model’s scores dropped.
*Updated Oct 23 to refresh the story and add an FAQ.*
