# No Model Clears 60% on Ivo’s New Contract Review Benchmark

> ContractBench tests whether AI makes the calls an experienced contracts lawyer would. The best score was 58.3%. The gaps matter more than the ranking.

By The Circle Desk | AI in Practice | Published 2026-10-23 | Updated 2026-10-23
Source: https://legaltechcircle.com/insights/no-model-clears-60-percent-ivo/

Can AI mark up a contract like an experienced lawyer? Not yet. On
ContractBench, a benchmark contract AI company Ivo and data lab micro1
published on October 13, 2026, all seven models tested scored below
60% on weighted attorney-defined criteria, with the best at 58.3% and
the top three within 1.2 points, according to [Ivo's research
report](https://www.ivo.ai/research/ivo-micro1-contractbench).
[Artificial Lawyer covered the
results](https://www.artificiallawyer.com/2026/10/23/grok-top-for-contracts-al-on-hols-harvey-agents/)
on October 23, 2026.

## How Did Ivo Test Contract Review?

ContractBench has 350 tasks across 35 scenarios and five agreement
types: master services agreements, IP licenses, research partnerships,
commercial leases and marketing agreements. Thirty scenarios use U.S.
law and five use UK law. Half the tasks are a first pass on a clean
draft. The other half ask the model to respond to a counterparty's
markup.

Attorneys wrote the rubrics after doing each redline themselves. Every
rubric item asks whether the reviewer took the right action on a
clause: edit it, leave it alone, comment, or escalate it to someone
with authority. High-priority items carry the most weight. For
first-pass tasks, five attorneys independently reviewed the same
contract, so models were scored against five reasonable answers
instead of one.

Keep one design choice in mind as you read the results. The rubrics
are attorney-written, but the pass or fail grading was done by three
AI judges from different providers, with a majority deciding. Ivo
reports the judges agreed unanimously on 83.4% of items.

## Where Did the Models Fall Short?

The top three were close. Grok 4.7 scored 58.3%, Opus 5.5 58.1% and
Kimi K3 57.1%, per the [micro1 results
page](https://www.micro1.ai/benchmark/ivo-micro1-contract-bench).
Gemini 3.8 Flash and GLM-5.3 followed at about 55%, then DeepSeek V4
Pro at 48.4% and GPT-6 Astra at 45.6%. Completion rates varied: Opus
5.5 completed 80.8% of its runs, with most failures caused by hitting
the output limit, while Grok finished 99.5% and Kimi all of them. Ivo
says Kimi cost 38% less per completed run than Grok.

Honestly, the ranking is the least interesting part. On counterparty
markups, every model passed more than 96% of items where the right
move was to leave untouched text untouched, but on items that called
for a new edit, scores fell to between 15.1% and 36.5%. Six of seven
models scored lowest on the issues attorneys rated most important.

Context was hard too. Every model scored 9 to 15 points lower when a
playbook rule had to be applied to the specific facts of the deal than
when the rule applied no matter what, and six of seven did worst with
no playbook at all. One surprise: GPT-6 Astra ranked last overall but
passed 70.8% of escalation items, more than twice any other model,
making it the best at recognizing when a call was above its authority.

And the models don't redline like lawyers. Attorneys made 63.5% of
their edits inline, while the best model managed 36.6%. On first-pass
reviews, attorneys averaged 125 edits per task at 93 characters each;
models averaged 41 edits at 300 characters. Any lawyer who has
received a "markup" that's really a rewrite will recognize the
problem.

## How Much Weight Should Buyers Give a Vendor's Benchmark?

We'd treat it as a strong signal with two caveats. First, Ivo sells
contract review AI, so it has an interest in showing that raw models
fall short. The benchmark tests base models through one shared test
setup; it does not test Ivo's product or any other vendor's. Second,
AI judges did the grading. The high agreement rate helps, but
attorneys did not score every answer.

Even so, the weak spots are specific enough to test on your own
contracts: high-priority clauses, deal-specific judgment, counterparty
markups, and edits that rewrite a clause when a word would do.

## What Should Your Team Do With These Results?

If you run a large in-house team using AI for first-pass review,
spot-check the high-priority clauses hardest, because that is where
these models scored lowest, and ask vendors which model sits
underneath and whether they have tested it against something like
this. Haven't written down your positions and fallbacks yet? Playbooks
helped nearly every model, so that work comes before buying a tool
that depends on them. Small teams and solo counsel should be most
careful with counterparty markups, where every model's scores dropped.
And if you were hoping for a single winner to standardize on, the top
three sit within 1.2 points of each other.

We rate ContractBench **Real**: the method is careful and the gaps it
exposes are the ones buyers should test for. Discount the leaderboard,
and remember that the best model met fewer than 60% of weighted
attorney criteria.

## FAQ

**What is ContractBench?**

A benchmark from Ivo and micro1, published October 13, 2026, testing
whether AI models make the same judgment calls as experienced
attorneys reviewing and negotiating contracts, per [Ivo's
report](https://www.ivo.ai/research/ivo-micro1-contractbench).

**What was the top score?**

58.3%, on weighted attorney-defined criteria, per the October 13, 2026 report.

**Did playbooks help?**

Yes. Playbooks helped nearly every model in the test.

**Where did models struggle most?**

On high-priority clauses and on counterparty markups, where every
model's scores dropped.

*Updated Oct 23 to refresh the story and add an FAQ.*

---
Legal Tech Circle (https://legaltechcircle.com/). News and analysis, not legal advice.