Steel.dev®

AI Agent Benchmark Registry

Explore an AI agent eval registry and benchmark leaderboard covering web navigation, coding, desktop control, tool use, deep research, and general reasoning. Compare evaluation suites, tests, frameworks, tasks, evaluators, top scores, and benchmark scope in one place.

How to read this registry

Compare results only when task scope and evaluation method are reasonably comparable. Reproducible suites like WebArena are easier to rerun, while live-web evals like WebVoyager better capture production drift. Start with the category routes for web navigation, coding, and tool use before comparing leaderboard numbers across very different evaluation suites. If you want a single place to browse reported scores across many benchmarks, jump to the Benchmark Index.

REGISTRY

ALL RESEARCH WEB NAV DESKTOP CODING TOOL USE GENERAL SPECIALIZED

Desktop control benchmarks

Desktop control benchmarks matter when an agent has to operate a full graphical environment instead of a single browser tab or an API-only sandbox. Strong desktop evaluations cover opening applications, managing files, switching windows, entering data, and coordinating work across local tools and the web. They are especially useful for understanding whether a system can recover from timing issues, noisy interfaces, and partial observability. A good place to start is OSWorld, then compare it with platform-focused suites such as Windows Agent Arena. If your workflow lives mostly in the browser, the web navigation benchmarks are often a better first filter.

Desktop control benchmark - Self-hosted

Benchmark By xLang AI

369 cross-application desktop tasks across Ubuntu, Windows, and macOS. Covers Chrome, LibreOffice, VS Code, and more. Execution-based evaluation. Agents still well below the human baseline of 72%.

Evaluation Method: Execution-based
Top Model Score: 66.2%
Human Score: 72.4%
Task Count: 369

View OSWorld benchmark paper OSWorld GitHub repository

Desktop control benchmark - Self-hosted

Benchmark By AgentSea

Cross-platform desktop benchmark covering macOS, Windows, and Ubuntu with 2,000+ tasks. Focuses on real-world app interactions and long-horizon task completion.

Evaluation Method: Execution-based
Top Model Score: ~40%
Human Score: N/A
Task Count: 2,000+

View OSUniverse benchmark paper OSUniverse GitHub repository

Desktop control benchmark - Self-hosted

Benchmark By Show Lab

macOS-specific benchmark with 369 tasks spanning system preferences, Finder, Safari, and productivity apps. Complements OSWorld with platform-specific depth.

Evaluation Method: Execution-based
Top Model Score: ~35%
Human Score: ~72%
Task Count: 369

View macOSWorld benchmark paper macOSWorld GitHub repository

Desktop control benchmark - Self-hosted

Windows Agent Arena

Benchmark By Microsoft

154 tasks across real Windows 11 applications running in Azure VMs. Tests document editing, file management, system settings, and browser tasks. Full reproducibility via cloud snapshots.

Evaluation Method: Programmatic
Top Model Score: 19.5%
Human Score: 74.5%
Task Count: 154

View Windows Agent Arena benchmark paper Windows Agent Arena GitHub repository

Desktop control benchmark - Self-hosted

Benchmark By Google DeepMind

116 tasks across 20 real Android apps in a live emulated environment. Functional evaluation without cached states. Tests agents on real apps including Gmail, Chrome, and Settings.

Evaluation Method: Functional
Top Model Score: ~30%
Human Score: ~88%
Task Count: 116

View AndroidWorld benchmark paper AndroidWorld GitHub repository

Desktop control benchmark - Self-hosted

Benchmark By X-LANCE

A gym environment for mobile UI interaction built on Android emulator. Provides step-level rewards for fine-grained evaluation of touch-based agent interaction.

Evaluation Method: Step reward
Top Model Score: N/A
Human Score: N/A
Task Count: ~70 tasks

View Mobile-Env benchmark paper Mobile-Env GitHub repository

MISSING A BENCHMARK? OPEN A PR ON GITHUB TO ADD IT TO THE REGISTRY.

What is an AI agent benchmark?

An AI agent benchmark, eval, or evaluation suite is a structured way to test how well an agent completes tasks in an environment, not just how well a model writes a plausible answer. Instead of grading one response, these tests look at sequences of actions across websites, codebases, tools, desktops, or research workflows. In practice, they measure whether the system can make progress, stay grounded, and reach the correct end state.

That is the main difference between an agent benchmark and a standard LLM eval. A classic LLM test asks whether the model produced the right answer to a prompt. An agent evaluation asks whether the system can plan, recover from mistakes, use the right tools, and complete a workflow under realistic constraints. Strong benchmark leaderboards often track not only accuracy, but also task success, reliability, latency, and cost.

Common methods include exact-match grading, executable test suites, environment-state checks, human review, and LLM-as-judge scoring for open-ended work. Each has tradeoffs in rigor, scalability, and realism. Self-hosted suites are easier to rerun and compare over time, while public-web or live-software evaluations better reflect drift and production messiness. The best way to evaluate AI agents is usually to combine both.

FAQ

What are AI agent benchmarks? [+]

AI agent benchmarks are evaluations that measure whether an agent can complete multi-step tasks in an environment such as a browser, terminal, desktop, or tool stack. Unlike single-prompt model tests, they focus on action quality, task completion, recovery from mistakes, and end-to-end execution.

What is an agent eval registry? [+]

An agent eval registry is a curated index of AI agent benchmarks, evaluations, leaderboards, test suites, and frameworks. Instead of covering just one benchmark family, it helps you compare multiple evaluation options across web navigation, coding, desktop control, and tool use in one place.

How do you evaluate AI agents? [+]

You evaluate AI agents by testing them on multi-step tasks in realistic environments and measuring whether they reach the correct end state. Strong agent evaluations usually track task success, evaluator design, reliability, cost, latency, and recovery from mistakes. The right eval framework depends on whether you care about browser use, coding, tool use, desktop control, or general reasoning.

What is the best benchmark for coding agents? [+]

There is no single best benchmark for every use case, but SWE-bench Verified is widely treated as the most trusted benchmark for coding agents because it uses real repository issues and executable test suites. Terminal-Bench is also useful when you want to evaluate autonomous agents in command-line and systems workflows.

How do browser agent benchmarks differ from coding benchmarks? [+]

Browser agent benchmarks evaluate interaction with websites, page state, navigation, and visual or DOM-grounded actions. Coding benchmarks evaluate repository understanding, file edits, tool use, debugging, and test execution. Compare WebVoyager with SWE-bench Verified to see how different the environments and failure modes are.

What does self-hosted mean in an agent benchmark? [+]

Self-hosted means the benchmark environment can be run in a controlled local or containerized setup instead of depending entirely on live public services. That usually improves reproducibility and evaluation stability, but may be less representative of the messy real web or production software. Benchmarks like WebArena and OSWorld are good examples.

Why do benchmark scores differ across evaluators? [+]

Benchmark scores differ because evaluators measure success in different ways. Some use exact match, some use executable test suites, some verify environment state, and others rely on human review or LLM judges. A score on one benchmark or evaluator is not directly comparable to a score produced by a different evaluation method. Use the How to read this registry note before comparing results.