Harvey LAB-AA is an independent benchmark that tests AI models on 120 private, real-world legal tasks across 24 practice areas, grading each one against a strict rubric of binary pass/fail criteria. Built by Artificial Analysis as an implementation of Harvey’s Legal Agent Benchmark (LAB), it measures whether AI agents can produce complete, professional-grade legal deliverables — not just answer legal trivia. As of its July 2026 launch, even the best-performing model leaves roughly 86% of tasks incomplete, showing that frontier legal AI still has a long way to go.
This article breaks down what Harvey LAB-AA measures, how the leaderboard looks today, what it costs to compete at the top, and why the benchmark matters for anyone evaluating AI agents for legal work.
What Is Harvey LAB-AA?
Harvey LAB-AA (Legal Agent Benchmark, Artificial Analysis implementation) is a rigorous evaluation designed to test how well AI agents handle authentic legal work — the kind performed daily inside law firms and corporate legal departments. Rather than testing narrow question-answering, Harvey LAB-AA assigns models complete legal deliverables, such as drafting a deal team report or redlining an arbitration agreement, and checks the output against a professional rubric.
The benchmark spans practice areas including corporate M&A, capital markets, tax, litigation, bankruptcy, commercial leasing, and employment law. Each of the 120 tasks in Harvey LAB-AA was built by Harvey’s legal team, giving the benchmark a level of domain authenticity that generic coding or reasoning benchmarks don’t attempt to replicate.
The headline metric is the all-pass rate: the percentage of tasks where a model satisfies every single criterion in the grading rubric. This is a deliberately unforgiving standard. In real legal practice, a document that’s 90% correct but misses one material clause isn’t “mostly done” — it’s a liability. Harvey LAB-AA’s all-pass grading reflects that reality by giving no partial credit.
How Does Harvey LAB-AA Work?
The 120-Task Private Test Set
Harvey LAB-AA runs models against a private set of 120 tasks spanning 24 practice areas. Each task includes reference materials — contracts, memos, financial filings, and other source documents — that the model must read, analyze, and act on to produce the requested deliverable. Example tasks include:
- A change-of-control analysis across a target company’s material contracts in an M&A deal
- A deposition outline for litigation preparation
- An arbitration agreement redline
- M&A disclosure schedules
- A commercial lease review
Models are run on Artificial Analysis’s own agent harness, called Stirrup, which allows for features like context compaction so that long, document-heavy tasks don’t simply fail when they exceed a model’s context window.
All-Pass Rate vs. Criterion Pass Rate: What’s the Difference?
All-pass rate measures the share of tasks where every rubric criterion is satisfied, while criterion pass rate measures the share of individual criteria passed across all tasks. The gap between these two numbers reveals a lot about the current state of legal AI: most models pass a majority of individual criteria but rarely satisfy every requirement in a given task simultaneously.
That gap is stark in the Harvey LAB-AA results. Four models score above 90% on criterion pass rate, yet the top all-pass rate at launch is just 14.2%. In other words, models are getting most of the details right most of the time — but “most” isn’t good enough for a document that needs to be fully correct before a client, court, or counterparty sees it.
Harvey LAB-AA Results: Which AI Models Score Best?
Top Performers by All-Pass Rate
At launch, Claude Fable 5 (run in max mode, with a fallback to Claude Opus 4.8) leads the Harvey LAB-AA leaderboard with a 14.2% all-pass rate — nearly double the next tier of models. Claude Opus 4.8 (max) and GLM-5.2 (max) tie for second place at 7.5%, followed by MiniMax-M3 at 6.7% and Claude Sonnet 5 at 5.0%.
On criterion pass rate, the picture is a little more encouraging: Claude Fable 5 leads at 93.6%, with Claude Opus 4.8 (91.1%), GLM-5.2 (91.0%), and Claude Sonnet 5 (90.1%) all clearing the 90% bar. Still, 13 of the 28 models evaluated at Harvey LAB-AA’s launch fully pass zero tasks — a reminder that strong performance on individual criteria doesn’t guarantee a usable, complete deliverable.
Harvey LAB-AA Model Comparison
| Model | All-Pass Rate | Criterion Pass Rate | Cost per Task | Output Tokens per Task | Time per Task |
|---|---|---|---|---|---|
| Claude Fable 5 (max, w/ fallback) | 14.2% | 93.6% | ~$18.9 | ~117k | ~16.9 min |
| Claude Opus 4.8 (max) | 7.5% | 91.1% | ~$8.2 | ~111k | ~18.5 min |
| GLM-5.2 (max) | 7.5% | 91.0% | ~$1.3 | ~78k | ~5.0 min |
| MiniMax-M3 | 6.7% | — | — | — | — |
| Claude Sonnet 5 | 5.0% | 90.1% | ~$11.8 | ~179k | ~22.8 min |
This comparison table highlights a core tension in Harvey LAB-AA: the model with the best all-pass rate is also the most expensive to run, while a model like GLM-5.2 nearly matches Claude Opus 4.8’s score for a fraction of the cost and in a fraction of the time.
Why Is Frontier Legal AI Still Far From Solved?
Frontier legal AI remains far from solved because even the top-scoring model on Harvey LAB-AA fails to fully complete about 86% of professional legal tasks. This isn’t a case of models producing obviously wrong answers — it’s a case of models producing outputs that are close but not fully compliant with the exacting standards real legal work demands.
A few factors drive this gap:
- Rubric strictness. All-pass grading requires every criterion to be met, so a single missed clause, citation, or formatting requirement fails the entire task.
- Document complexity. Legal tasks in Harvey LAB-AA often require synthesizing information across a dozen or more reference documents.
- Domain-specific precision. Legal deliverables carry professional and financial consequences, so the acceptable margin for error is much smaller than in general-purpose writing or coding tasks.
This is a useful corrective for anyone assuming that strong benchmark scores elsewhere (coding, general reasoning) automatically translate to law-firm-ready output. Harvey LAB-AA suggests they don’t — at least not yet.
Cost and Token Usage: What It Really Takes to Win Harvey LAB-AA
Topping the Harvey LAB-AA leaderboard isn’t cheap. Claude Fable 5 leads all-pass rate at 14.2% but costs roughly $18.9 per task to run — the most expensive model in the evaluation. Claude Sonnet 5 follows at about $11.8 per task, while Claude Opus 4.8 costs around $8.2 per task. GLM-5.2 (max) matches Opus 4.8’s all-pass score at only about 15% of the cost — roughly $1.3 per task.
Cost per task varies enormously across the full field of models evaluated on Harvey LAB-AA — by close to 950x from cheapest to most expensive. At the low end, Gemini 3.1 Flash-Lite passes 31.1% of individual criteria for about $0.02 per task, illustrating that cost and criterion-level competence don’t move in lockstep.
Token usage tracks closely with performance. All-pass rate on Harvey LAB-AA is generally tied to how much work a model does per task:
- Claude Fable 5 generates about 117,000 output tokens per task to reach its 14.2% all-pass rate.
- Claude Opus 4.8 and GLM-5.2 use roughly 111,000 and 78,000 output tokens respectively for their tied 7.5% score.
- Claude Sonnet 5 generates the most output of any model evaluated — about 179,000 tokens per task — despite scoring lower on all-pass rate than the top three.
Speed and Turns: How Long Do AI Agents Take on Legal Tasks?
Stronger performance on Harvey LAB-AA generally comes with longer processing time. Claude Fable 5 averages about 16.9 minutes per task, Claude Opus 4.8 about 18.5 minutes, and Claude Sonnet 5 about 22.8 minutes. GLM-5.2 is the notable exception, matching Claude Opus 4.8’s all-pass score in just about 5.0 minutes per task — making it the most time-efficient of the top performers.
Agent behavior also varies widely in how many turns — individual actions, tool calls, or iteration cycles — a model takes to complete a task:
- Claude Fable 5 averages around 64 turns per task.
- Claude Opus 4.8, GLM-5.2, and MiniMax-M3 all fall in a similar 56–64 turn range.
- Claude Sonnet 5 runs the longest agentic loops of any model tested, at roughly 161 turns per task — about 2.5 times more than Claude Fable 5.
More turns don’t necessarily mean better results; Claude Sonnet 5’s longer loops didn’t translate into a higher all-pass rate than the top three models on Harvey LAB-AA.
How Harvey LAB-AA Differs From Harvey’s Original LAB
Harvey LAB-AA is Artificial Analysis’s independent reimplementation of Harvey’s original Legal Agent Benchmark, not a direct rerun of it. The key differences include:
- Models run on Artificial Analysis’s open-source Stirrup agent harness, which supports context compaction instead of failing outright when a task exceeds context limits, using simplified agent and judge prompts.
- Harvey’s proprietary tools and document-generation skill scripts (for formats like .pptx and .docx) are excluded; Harvey LAB-AA instead provides a general-purpose code execution tool to test raw model capability.
- Deliverables must match the exact filename specified in the task — there’s no fuzzy matching for incorrectly named files.
- Grading is performed by a single Gemini 3.1 Pro judge model, which Artificial Analysis calibrated against a panel of frontier models to ensure grading consistency.
These design choices make Harvey LAB-AA a stricter, more standardized test of underlying model capability, separate from the convenience tooling that might be layered on top in a commercial legal AI product.
FAQs About Harvey LAB-AA
What does Harvey LAB-AA measure? Harvey LAB-AA measures how well AI models complete authentic legal deliverables across 24 practice areas, using an all-pass grading standard where every rubric criterion must be satisfied for a task to count as passed.
Which model currently leads Harvey LAB-AA? Claude Fable 5, run in max mode with a fallback to Claude Opus 4.8, leads Harvey LAB-AA with a 14.2% all-pass rate as of its July 2026 launch.
Is Harvey LAB-AA the same as Harvey’s original benchmark? No. Harvey LAB-AA is Artificial Analysis’s independent implementation of Harvey’s Legal Agent Benchmark, using a different agent harness, simplified prompts, stricter filename matching, and a single calibrated judge model rather than Harvey’s original evaluation setup.
Why do models pass so few tasks on Harvey LAB-AA despite high criterion pass rates? Because Harvey LAB-AA uses all-pass grading with no partial credit, a model can satisfy 90% or more of individual rubric criteria across a task set and still fail most individual tasks if even one criterion is missed per task.
Where can I see live Harvey LAB-AA results? Up-to-date results are published on the Harvey LAB-AA evaluation page on Artificial Analysis’s website, which updates as new models are released and tested.
Key Takeaways
- Harvey LAB-AA tests AI agents on 120 private legal tasks across 24 practice areas, using strict all-pass grading with no partial credit.
- Claude Fable 5 leads the launch leaderboard at a 14.2% all-pass rate, nearly double the next tier of models.
- Even the top model on Harvey LAB-AA leaves about 86% of tasks incomplete, underscoring that frontier legal AI is far from solved.
- Higher all-pass scores on Harvey LAB-AA correlate with higher cost, more output tokens, and longer processing time — though GLM-5.2 breaks that pattern with strong efficiency.
- Harvey LAB-AA differs meaningfully from Harvey’s original benchmark in harness, tooling, filename matching, and grading methodology.