Steel.dev®

AI Agent Benchmark Registry

Steel.dev Logo

Explore an AI agent eval registry and benchmark leaderboard covering web navigation, coding, desktop control, tool use, deep research, and general reasoning. Compare evaluation suites, tests, frameworks, tasks, evaluators, top scores, and benchmark scope in one place.

How to read this registry

Compare results only when task scope and evaluation method are reasonably comparable. Reproducible suites like WebArena are easier to rerun, while live-web evals like WebVoyager better capture production drift. Start with the category routes for web navigation, coding, and tool use before comparing leaderboard numbers across very different evaluation suites. If you want a single place to browse reported scores across many benchmarks, jump to the Benchmark Index.

REGISTRY
ALL RESEARCH WEB NAV DESKTOP CODING TOOL USE GENERAL SPECIALIZED

Specialized agent benchmarks

Specialized agent benchmarks focus on narrow but commercially important workflows such as scientific work, social simulation, medical tasks, customer support, and other domain-heavy settings. These evaluations are useful because broad leaderboard strength often hides uneven performance once the agent has to follow domain rules, preserve citations, or navigate unusual interfaces. A specialized eval should define a concrete environment, clear artifacts of success, and constraints that actually resemble the target workflow. To get oriented, look at MedAgentBench or Sotopia, then compare those results with the broader general reasoning benchmarks before drawing product conclusions.
Specialized agent benchmark - Public
Sotopia
Benchmark By Sotopia Lab

Social intelligence benchmark placing agents in realistic social scenarios. Evaluates believability, social goal completion, relationship management, and secret keeping across 11 social dimensions.

Top Model Score
~7.6/10
GPT-4
Human Score
~8.3/10
View Sotopia benchmark paper Sotopia GitHub repository
Specialized agent benchmark - Public
AgentHarm
Benchmark By AIEvals

Safety red-teaming benchmark with 440 harmful agent tasks across 11 categories. Tests whether agent frameworks allow harmful behaviors — jailbreaking, weapon synthesis, fraud, and more.

Top Model Score
N/A
N/A (safety eval)
Human Score
N/A
View AgentHarm benchmark paper AgentHarm GitHub repository
Specialized agent benchmark - Public
MedAgentBench
Benchmark By Stanford

300 clinical tasks across 10 medical categories using real EHR data. Tests agents on diagnosis reasoning, treatment planning, and medical record navigation in realistic hospital environments.

Top Model Score
~77%
o1-preview
Human Score
N/A
View MedAgentBench benchmark paper MedAgentBench GitHub repository
Specialized agent benchmark - Public
MultiAgentBench
Benchmark By UIUC

Evaluates both cooperative and competitive multi-agent systems. Tasks include collaborative problem-solving and adversarial games — measures emergent coordination and strategic behavior.

Top Model Score
N/A
GPT-4 (multi-agent)
Human Score
N/A
View MultiAgentBench benchmark paper MultiAgentBench GitHub repository
Specialized agent benchmark - Self-hosted
FORTRESS

Cybersecurity benchmark testing agents on capture-the-flag style challenges. Covers reverse engineering, web exploitation, and cryptography. Designed to stress-test autonomous offensive security agents.

Top Model Score
~35%
o3
Human Score
N/A
View FORTRESS benchmark paper
Specialized agent benchmark - Public
Vending-Bench

Transaction and inventory reasoning benchmark. Agents manage a virtual vending machine over many turns — testing whether models understand real-world economics, stock levels, and pricing logic.

Top Model Score
~62%
Claude 3.5 Sonnet
Human Score
N/A
View Vending-Bench benchmark paper
Specialized agent benchmark - Public
CharacterEval

Role-playing and character consistency benchmark. Evaluates agents on maintaining persona fidelity, character knowledge accuracy, and in-character behavior across long conversations.

Top Model Score
~75%
GPT-4
Human Score
N/A
View CharacterEval benchmark paper CharacterEval GitHub repository
Specialized agent benchmark - Public
Bullshit Benchmark
Benchmark By Peter Gostev

Tests model resistance to confidently stated falsehoods in prompts. Evaluates whether agents can identify and reject plausible-sounding but incorrect premises before acting on them.

Top Model Score
N/A
o3
Human Score
N/A
View Bullshit Benchmark benchmark paper Bullshit Benchmark GitHub repository

MISSING A BENCHMARK? OPEN A PR ON GITHUB TO ADD IT TO THE REGISTRY.

What is an AI agent benchmark?

An AI agent benchmark, eval, or evaluation suite is a structured way to test how well an agent completes tasks in an environment, not just how well a model writes a plausible answer. Instead of grading one response, these tests look at sequences of actions across websites, codebases, tools, desktops, or research workflows. In practice, they measure whether the system can make progress, stay grounded, and reach the correct end state.

That is the main difference between an agent benchmark and a standard LLM eval. A classic LLM test asks whether the model produced the right answer to a prompt. An agent evaluation asks whether the system can plan, recover from mistakes, use the right tools, and complete a workflow under realistic constraints. Strong benchmark leaderboards often track not only accuracy, but also task success, reliability, latency, and cost.

Common methods include exact-match grading, executable test suites, environment-state checks, human review, and LLM-as-judge scoring for open-ended work. Each has tradeoffs in rigor, scalability, and realism. Self-hosted suites are easier to rerun and compare over time, while public-web or live-software evaluations better reflect drift and production messiness. The best way to evaluate AI agents is usually to combine both.

FAQ
What are AI agent benchmarks? [+]
AI agent benchmarks are evaluations that measure whether an agent can complete multi-step tasks in an environment such as a browser, terminal, desktop, or tool stack. Unlike single-prompt model tests, they focus on action quality, task completion, recovery from mistakes, and end-to-end execution.
What is an agent eval registry? [+]
An agent eval registry is a curated index of AI agent benchmarks, evaluations, leaderboards, test suites, and frameworks. Instead of covering just one benchmark family, it helps you compare multiple evaluation options across web navigation, coding, desktop control, and tool use in one place.
How do you evaluate AI agents? [+]
You evaluate AI agents by testing them on multi-step tasks in realistic environments and measuring whether they reach the correct end state. Strong agent evaluations usually track task success, evaluator design, reliability, cost, latency, and recovery from mistakes. The right eval framework depends on whether you care about browser use, coding, tool use, desktop control, or general reasoning.
What is the best benchmark for coding agents? [+]
There is no single best benchmark for every use case, but SWE-bench Verified is widely treated as the most trusted benchmark for coding agents because it uses real repository issues and executable test suites. Terminal-Bench is also useful when you want to evaluate autonomous agents in command-line and systems workflows.
How do browser agent benchmarks differ from coding benchmarks? [+]
Browser agent benchmarks evaluate interaction with websites, page state, navigation, and visual or DOM-grounded actions. Coding benchmarks evaluate repository understanding, file edits, tool use, debugging, and test execution. Compare WebVoyager with SWE-bench Verified to see how different the environments and failure modes are.
What does self-hosted mean in an agent benchmark? [+]
Self-hosted means the benchmark environment can be run in a controlled local or containerized setup instead of depending entirely on live public services. That usually improves reproducibility and evaluation stability, but may be less representative of the messy real web or production software. Benchmarks like WebArena and OSWorld are good examples.
Why do benchmark scores differ across evaluators? [+]
Benchmark scores differ because evaluators measure success in different ways. Some use exact match, some use executable test suites, some verify environment state, and others rely on human review or LLM judges. A score on one benchmark or evaluator is not directly comparable to a score produced by a different evaluation method. Use the How to read this registry note before comparing results.