Scores
How model sizes and weights are classified
Size classes follow the Artificial Analysis categories: Tiny ≤4B; Small >4–40B; Medium >40–150B; Large >150B total parameters. Active MoE parameters are reported separately.
When parameters are unavailable, we use standard uncached token prices with a 3:1 input/output blend: Small <$0.15/M, Medium $0.15–$1/M, Large >$1/M. “Price proxy” is a pricing category, not a parameter estimate. “Estimated” marks a provisional classification and remains included in filters. “Unclassified” results await model research and match only the all and mixed / uncertain weight filters.
Harnesses inherit the evaluated model stack's class. Multi-model systems use the largest component category as an approximation. Open weights requires released weights for every required model; open harness code alone does not qualify. Unverified backbones and unavailable evaluated checkpoints are marked uncertain.
Results compare evaluated systems, including their tools, compute budgets, and context strategies. Size categories do not guarantee hardware compatibility. Suggest a correction ↗
Results
Agent scope · 22 systems| Rank | System / submission | Score ↓ | Model class | Reported | Source |
|---|---|---|---|---|---|
| 1 | browser-control (Fable 5)NewOm Labs · HarnessEvaluation & model detailsShell-native CDP harness driving Fable 5; 610/615 after four documented current-impossible exclusions; GPT-5.5 Alumnium-vanilla judge. Self-reported. Published runner names claude-fable-5 through Claude Code with the public browser-control CDP tool. GPT-5.5 is an external scoring judge and is excluded. The author warns that the evaluated Fable 5 availability changed after the run; released harness code does not imply open model weights. Harness code: open. Price proxy: the size class comes from the standard token price of Claude Fable 5; parameters are undisclosed. Claude Fable 5Closed weights · Large · Price proxy Token price: $10 input / $50 output per million. Blended 3:1: $20/M. Undisclosed parameters; size uses standard uncached token pricing. Tool charges excluded. Checked 2026-10-10 · high confidence | 99.19% | Closed weightsLargePrice proxy | 2026-06 | Source ↗Repository ↗ |
| 2 | AlumniumNewAlumnium · HarnessEvaluation & model detailsClaude Code orchestrating Alumnium MCP with GPT-5 Nano + Selenium; accessibility-tree parsing with visual reasoning. March 2026 setup uses Claude Code/Sonnet 4.6 as orchestrator and Alumnium MCP 0.18/GPT-5 nano as browser subagent. Both are required runtime models; the stack uses the larger category. GPT-5 only scores completed trajectories and is excluded. Harness code: open. Price proxy: the size class comes from the standard token price of Claude Sonnet 4.6; parameters are undisclosed. Size reflects the largest of 2 models; parameters are not summed. Claude Sonnet 4.6Closed weights · Large · Price proxy Token price: $3 input / $15 output per million. Blended 3:1: $6/M. Undisclosed parameters; size uses standard uncached token pricing. Tool charges excluded. Checked 2026-10-10 · high confidence GPT-5 nanoClosed weights · Small · Price proxy Token price: $0.05 input / $0.4 output per million. Blended 3:1: $0.1375/M. Hosted OpenAI model; total parameters undisclosed. Standard uncached $0.05 input / $0.40 output per 1M tokens gives a $0.1375 blend, below the Small cutoff. Alumnium uses it as the browser subagent. Checked 2026-10-10 · high confidence | 98.5% | Closed weightsLargePrice proxyLargest component | 2026-03 | Source ↗Repository ↗ |
| 3 | Surfer 2H Company · SystemEvaluation & model detailsSystem-level orchestration with submitter-defined setup details. Table 1’s WebVoyager stack is o3 orchestration, Sonnet 4.5 policy, GPT-4.1 validation and Holo1.5-7B localization. Validation feeds retries, so it is runtime. This differs from the paper’s WebArena configuration. Mixed open/closed weights; release of the exact Surfer 2 harness was not verified. Harness code: uncertain. Price proxy: the size class comes from the standard token price of o3, Claude Sonnet 4.5, GPT-4.1; parameters are undisclosed. Size reflects the largest of 4 models; parameters are not summed. o3Closed weights · Large · Price proxy Token price: $2 input / $8 output per million. Blended 3:1: $3.5/M. Parameter count is undisclosed. Large is a price proxy from standard uncached API text rates; it does not assert a physical parameter size. Tool fees and cached, batch, flex, and priority rates are excluded. Checked 2026-10-10 · high confidence Claude Sonnet 4.5Closed weights · Large · Price proxy Token price: $3 input / $15 output per million. Blended 3:1: $6/M. Closed hosted model with undisclosed parameter count. Published standard uncached first-party text prices are $3 input/$15 output per million tokens, giving a $6/M blend and Large price-proxy category. Cache, batch, and tool charges are excluded. This uses the standard context tier (up to 200K input tokens), not long-context premium pricing. Checked 2026-10-10 · high confidence GPT-4.1Closed weights · Large · Price proxy Token price: $2 input / $8 output per million. Blended 3:1: $3.5/M. Hosted OpenAI GPT-4.1 family with published 2025-04-14 snapshot; parameters undisclosed. Standard uncached $2/$8 token prices give a $3.50 blend. Surfer 2 uses it for validation inside the action/retry loop. Checked 2026-10-10 · high confidence Holo1.5-7BOpen weights · Small · Published size · 8.292B total Public ungated Apache-2.0 checkpoint with four downloadable safetensors shards. Inventory reports 8,292,166,656 total parameters including the vision components; the 7B product name describes the language-model scale. Use the inventory total for classification. Checked 2026-10-10 · high confidence | 97.1% | Mixed weightsLargePrice proxyLargest component | 2025-10 | Source ↗ |
| 4 | MagnitudeMagnitude · HarnessEvaluation & model detailsOpen-source architecture utilizing a modular agentic stack. The 93.9% report and public runner identify Claude Sonnet 4; code pins claude-sonnet-4-20250514 through Claude Code. Original magnitude.run/webvoyager redirects to a missing page; the author’s evaluation repository supplies the recovered evidence. Harness code: open. Price proxy: the size class comes from the standard token price of Claude Sonnet 4 (20250514); parameters are undisclosed. Claude Sonnet 4 (20250514)Closed weights · Large · Price proxy Token price: $3 input / $15 output per million. Blended 3:1: $6/M. Magnitude runner pins claude-sonnet-4-20250514. Parameters undisclosed. Standard uncached $3/$15 per 1M tokens gives a $6 blend. Claude Code is the provider wrapper, not another separately counted model. Checked 2026-10-10 · high confidence | 93.9% | Closed weightsLargePrice proxy | 2025-07 | Source ↗Repository ↗ |
| 5 | Surfer-H + Holo1H Company · HarnessEvaluation & model detailsMulti-modal action kernels integrated via H-Company research. The 92.2% Table 5 run uses Holo1-7B policy/localization plus GPT-4o validation with up to ten attempts. The validator supplies retry feedback and must be included; the all-Holo variant scores differently. GPT-4o snapshot is unpinned. Open CLI code and Holo weights do not make this mixed stack all-open. Harness code: open. Estimated for GPT-4o (historical snapshot unspecified): no verified parameter count or standard token price, so the size class is provisional. Size reflects the largest of 2 models; parameters are not summed.
Holo1-7BOpen weights · Small · Published size · 8.292B total Public ungated Apache-2.0 checkpoint with four downloadable safetensors shards. Inventory reports 8,292,166,656 total parameters including the vision components; the 7B product name describes the language-model scale. Use the inventory total for classification. Checked 2026-10-10 · high confidence GPT-4o (historical snapshot unspecified)Closed weights · Large · Estimated Reports/configurations name GPT-4o without a verifiable snapshot or use an unpinned Azure deployment alias. Keep separate from May, August and November 2024 checkpoints. Large is provisional from known GPT-4o standard price tiers; no exact tariff or parameter count is assigned to these historical runs.
Checked 2026-10-10 · low confidence | 92.2% | Mixed weightsLargeEstimatedLargest component | 2025-06 | Source ↗Repository ↗ |
| 6 | browser-control (Opus 4.8)Newbrowser-control · HarnessEvaluation & model detailsShell-native CDP harness driving Claude Opus 4.8; 556/615 on a model-specific layer with 47 prompt edits; GPT-5.5 Alumnium-vanilla judge. Self-reported. Published runner names claude-opus-4-8 through Claude Code with public browser-control tooling. GPT-5.5 evaluates completed runs and is excluded from the runtime. Open harness code is distinct from closed Claude weights. Harness code: open. Price proxy: the size class comes from the standard token price of Claude Opus 4.8; parameters are undisclosed. Claude Opus 4.8Closed weights · Large · Price proxy Token price: $5 input / $25 output per million. Blended 3:1: $10/M. Undisclosed parameters; size uses standard uncached token pricing. Tool charges excluded. Checked 2026-10-10 · high confidence | 90.41% | Closed weightsLargePrice proxy | 2026-06 | Source ↗Repository ↗ |
| 7 | BrowserableBrowserable · HarnessEvaluation & model detailsFine-tuned browser control models within a commercial framework. Author report specifies Gemini 2.0 Flash primary with GPT-4o and Claude 3.5 Sonnet fallbacks for rate limits. Include the available fallback stack; invocation shares and historical snapshots are unspecified. The retrieved report does not establish custom released fine-tuned weights. Harness code: open. Estimated for GPT-4o (historical snapshot unspecified), Claude 3.5 Sonnet (historical unpinned snapshot): no verified parameter count or standard token price, so the size class is provisional. Size reflects the largest of 3 models; parameters are not summed. Gemini 2.0 FlashClosed weights · Medium · Price proxy Token price: $0.15 input / $0.6 output per million. Blended 3:1: $0.2625/M. Hosted Gemini 2.0 Flash; parameters undisclosed. Google Cloud publishes standard uncached text/image/video input $0.15 and text output $0.60 per 1M tokens, a $0.2625 blend. This is the verified Vertex tariff, not a claim about historical benchmark billing or free quotas; the current Developer API price page no longer lists this model.
Checked 2026-10-10 · medium confidence GPT-4o (historical snapshot unspecified)Closed weights · Large · Estimated Reports/configurations name GPT-4o without a verifiable snapshot or use an unpinned Azure deployment alias. Keep separate from May, August and November 2024 checkpoints. Large is provisional from known GPT-4o standard price tiers; no exact tariff or parameter count is assigned to these historical runs.
Checked 2026-10-10 · low confidence Claude 3.5 Sonnet (historical unpinned snapshot)Closed weights · Large · Estimated The source identifies Claude 3.5 Sonnet but not its June/October snapshot. Provisional Large follows this proprietary frontier family; parameters and an exact evaluated-version tariff are not asserted. Snapshot-specific records remain separate. Checked 2026-10-10 · low confidence | 90.4% | Closed weightsLargeEstimatedLargest component | 2025-04 | Source ↗Repository ↗ |
| 8 | Browser UseBrowser Use · HarnessEvaluation & model detailsMulti-step orchestration framework for open-source automation. December 2024 report names GPT-4o; public evaluation code routes Azure deployments named gpt-4o without a verifiable snapshot. Keep historical identity provisional. External scoring model is excluded. Public harness code does not imply open GPT-4o weights. Harness code: open. Estimated for GPT-4o (historical snapshot unspecified): no verified parameter count or standard token price, so the size class is provisional. GPT-4o (historical snapshot unspecified)Closed weights · Large · Estimated Reports/configurations name GPT-4o without a verifiable snapshot or use an unpinned Azure deployment alias. Keep separate from May, August and November 2024 checkpoints. Large is provisional from known GPT-4o standard price tiers; no exact tariff or parameter count is assigned to these historical runs.
Checked 2026-10-10 · low confidence | 89.1% | Closed weightsLargeEstimated | 2024-12 | Source ↗Repository ↗ |
| 9 | GLM-5V-TurboNewZ.ai · ModelEvaluation & model detailsMultimodal vision model optimized for GUI automation and coding tasks. Known Z.ai hosted model; parameter total and released evaluated weights are unverified. Large comes from the serving provider’s published standard token-price proxy, not a parameter estimate or another GLM model’s architecture. Harness code: not separately classified. Price proxy: the size class comes from the standard token price of GLM-5V-Turbo; parameters are undisclosed.
GLM-5V-TurboUncertain weights · Large · Price proxy Token price: $1.2 input / $4 output per million. Blended 3:1: $1.9/M. Exact multimodal API model is documented, but its parameter count and evaluated weights are not verified. The Z.AI pricing page currently omits this model; the serving Vercel AI Gateway directly lists its Z.AI route at standard $1.20 input/$4 output per million tokens (200K context), a $1.90/M blend. This is a provider price proxy, not a claim about parameter size. The exact zai-org checkpoint lookup returns HTTP 401 and the author search has no 5V release.
Checked 2026-10-10 · medium confidence | 88.5% | Uncertain weightsLargePrice proxy | 2026-04 | Source ↗ |
| 10 | Agent KuraKura · SystemEvaluation & model details602/643 tasks (41 removed for invalid/auth issues); reported on trykura.com. Kura describes an o1 initial planner plus GPT-4o loop planner, executor, critic and final critic. Historical snapshots are unpinned; the currently served primary pages do not prove a November 2024 checkpoint. Steel retrieved the JavaScript-rendered pages. Public benchmark assets do not establish release of the full agent implementation. Harness code: uncertain. Estimated for OpenAI o1 family (Kura historical run), GPT-4o (historical snapshot unspecified): no verified parameter count or standard token price, so the size class is provisional. Size reflects the largest of 2 models; parameters are not summed. OpenAI o1 family (Kura historical run)Closed weights · Large · Estimated Kura names OpenAI o1 for its initial planner without a snapshot. The historical timing may indicate o1-preview, but this is unconfirmed and must not be replaced silently by the later full o1 checkpoint. Provisional Large follows the o1 frontier reasoning family’s known price class; exact parameters and run-specific tariff are unavailable.
Checked 2026-10-10 · low confidence GPT-4o (historical snapshot unspecified)Closed weights · Large · Estimated Reports/configurations name GPT-4o without a verifiable snapshot or use an unpinned Azure deployment alias. Keep separate from May, August and November 2024 checkpoints. Large is provisional from known GPT-4o standard price tiers; no exact tariff or parameter count is assigned to these historical runs.
Checked 2026-10-10 · low confidence | 87.0% | Closed weightsLargeEstimatedLargest component | 2024-11 | Source ↗ |
| 10 | OperatorOpenAI · SystemEvaluation & model detailsNative browser integration using proprietary vision-control models. Original January 2025 Operator uses OpenAI’s computer-using agent. Keep it separate from later computer-use-preview/API snapshots and tariffs; original model scale remains a provisional estimate. Harness code: closed. Estimated for OpenAI original Computer-Using Agent (January 2025): no verified parameter count or standard token price, so the size class is provisional. OpenAI original Computer-Using Agent (January 2025)Closed weights · Large · Estimated The original Operator CUA combines GPT-4o vision with reinforcement-learned reasoning. Its parameter count and January evaluation token prices are undisclosed. Large is a provisional category informed by the GPT-4o-based frontier architecture and later computer-use-preview pricing, not a parameter estimate or assertion that the March API snapshot is identical. Do not substitute later o3-based Operator or computer-use-preview-2025-03-11 for this historical model.
Checked 2026-10-10 · medium confidence | 87% | Closed weightsLargeEstimated | 2025-01 | Source ↗ |
| 12 | NotteNotte · HarnessEvaluation & model detailsSelf-reported score from Notte's open-operator-evals harness (79.0% under LLM evaluation). The v1.3.3 report attributes 86.2% to Gemini 2.0 Flash on WebVoyager30, a 30-task subset run eight times, using self-reported completion. Configuration and handler set both reasoning and perception to Gemini; the default Llama helper is overridden. The published external LLM score differs; preserve the existing leaderboard result. Harness code: open. Price proxy: the size class comes from the standard token price of Gemini 2.0 Flash; parameters are undisclosed.
Gemini 2.0 FlashClosed weights · Medium · Price proxy Token price: $0.15 input / $0.6 output per million. Blended 3:1: $0.2625/M. Hosted Gemini 2.0 Flash; parameters undisclosed. Google Cloud publishes standard uncached text/image/video input $0.15 and text output $0.60 per 1M tokens, a $0.2625 blend. This is the verified Vertex tariff, not a claim about historical benchmark billing or free quotas; the current Developer API price page no longer lists this model.
Checked 2026-10-10 · medium confidence | 86.2% | Closed weightsMediumPrice proxy | 2025-04 | Source ↗Repository ↗ |
| 13 | Skyvern 2.0Skyvern · HarnessEvaluation & model detailsDOM-level reasoning coupled with real-time error-correction. Historical benchmark setup names GPT-4o and GPT-4o-mini as decision models and runs on Skyvern Cloud. The currently updated blog retains this setup, but does not pin GPT-4o’s snapshot. Public Skyvern harness code is separate from the closed hosted model weights. Harness code: open. Estimated for GPT-4o (historical snapshot unspecified): no verified parameter count or standard token price, so the size class is provisional. Size reflects the largest of 2 models; parameters are not summed. GPT-4o (historical snapshot unspecified)Closed weights · Large · Estimated Reports/configurations name GPT-4o without a verifiable snapshot or use an unpinned Azure deployment alias. Keep separate from May, August and November 2024 checkpoints. Large is provisional from known GPT-4o standard price tiers; no exact tariff or parameter count is assigned to these historical runs.
Checked 2026-10-10 · low confidence GPT-4o miniClosed weights · Medium · Price proxy Token price: $0.15 input / $0.6 output per million. Blended 3:1: $0.2625/M. Standard uncached input/output pricing gives a $0.2625/M blend: Medium price proxy. InfoAgent uses this hosted model for search snippets. Parameter count is undisclosed. Checked 2026-10-10 · high confidence | 85.85% | Closed weightsLargeEstimatedLargest component | 2025-01 | Source ↗Repository ↗ |
| 14 | Project MarinerGoogle · SystemEvaluation & model detailsGemini-powered reasoning with precise visual grounding. December 2024 source describes the 83.5% single-agent research prototype as built with Gemini 2.0. It does not identify Flash or disclose an independently priced model checkpoint. Model category remains a low-confidence estimate. Harness code: closed. Estimated for Project Mariner Gemini 2.0 prototype: no verified parameter count or standard token price, so the size class is provisional. Project Mariner Gemini 2.0 prototypeClosed weights · Large · Estimated Google identifies a December 2024 research prototype built with Gemini 2.0, without a public model checkpoint, parameter count or model-specific token price. It is not identified as Gemini 2.0 Flash. Large is a low-confidence frontier-agent prior; actual scale could fall in a smaller category. Checked 2026-10-10 · low confidence | 83.5% | Closed weightsLargeEstimated | 2024-12 | Source ↗ |
| 15 | Wuying-Browser-Agent-27BNewAlibaba Cloud · SystemEvaluation & model detailsAlibaba self-report: pass@1 on WebVoyager (643-task benchmark), 100 steps, custom browser-use with DOM and screenshots; the judge family also supplies RL rewards, with 96.4% agreement in a 500-trajectory human audit. The evaluated fine-tuned policy is identified by the primary paper; scoring judges and training teachers are excluded from the runtime model classification. Exact evaluated weight availability has not been verified in this refresh. Harness code: uncertain. Published size: from the parameter count of Wuying-Browser-Agent-27B. Wuying-Browser-Agent-27BUncertain weights · Small · Published size · 27B total Qwen3.5-27B-based trained browser policy; 27B is the published nominal parameter count. The paper describes an open-source release, but exact evaluated checkpoint files and harness release were not independently verified in this refresh. Checked 2026-10-11 · medium confidence | 80.6% | Uncertain weightsSmall | 2026-08-18 | Source ↗ |
| 16 | Agent-EEmergence AI · HarnessEvaluation & model detailsHierarchical planning modules within a multi-agent framework. The paper uses GPT-4 Turbo for planner and browser navigator; July 2024 README recommends stable gpt-4-turbo. Mapping to the 2024-04-09 stable snapshot follows that API alias, not an explicit evaluation snapshot log. Current repository examples using other models do not replace the historical setup. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-4 Turbo (2024-04-09); parameters are undisclosed.
GPT-4 Turbo (2024-04-09)Closed weights · Large · Price proxy Token price: $10 input / $30 output per million. Blended 3:1: $15/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. The stable Turbo snapshot is distinct from earlier previews. Checked 2026-10-10 · high confidence | 73.2% | Closed weightsLargePrice proxy | 2024-07 | Source ↗Repository ↗ |
| 17 | WebSightAcademic Research · HarnessEvaluation & model detailsNavigation system prioritizing visual-only perceptual inputs. Paper and agent code use GPT-4.1 mini planning/next-action reasoning plus the released WebSight-7B action checkpoint. This is a mixed-weight runtime, so the open action model alone does not make the submission open. Teachers and external scoring judges are excluded. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-4.1 mini; parameters are undisclosed. Size reflects the largest of 2 models; parameters are not summed.
WebSight-7BOpen weights · Small · Published size · 8.292B total The agent loads tanvirb/websight-7B, a merged UI-TARS-1.5-7B fine-tune. Public ungated Apache-2.0 repository has four full safetensors shards, in addition to archived adapters. Inventory total is 8,292,166,656 parameters including vision components.
Checked 2026-10-10 · high confidence GPT-4.1 miniClosed weights · Medium · Price proxy Token price: $0.4 input / $1.6 output per million. Blended 3:1: $0.7/M. Hosted OpenAI model; parameters undisclosed. Standard uncached $0.40/$1.60 token prices give a $0.70 blend. The WebSight code names the GPT-4.1 mini API alias for planning and next-action reasoning.
Checked 2026-10-10 · high confidence | 68% | Mixed weightsMediumPrice proxyLargest component | 2025-08 | Source ↗ |
| 18 | Runner H 0.1H Company · SystemEvaluation & model detailsFoundational agent architecture for general web interaction. Historical report identifies internal H-VLM 3B and H-LLM 2B plus unspecified external runtime models. Retain all three components; do not substitute later open Holo weights. GPT-4o is identified only for evaluation. Original source moved; author’s archived research update was retrieved. Harness code: uncertain. Estimated for Runner H 0.1 external model stack (undisclosed): no verified parameter count or standard token price, so the size class is provisional. Size reflects the largest of 3 models; parameters are not summed. Runner H H-VLM 3BUncertain weights · Tiny · Published size · 3B total H Company’s historical report specifies a 3B vision-language runtime model. This is the original internal H-VLM, not the later Holo1 release. No downloadable evaluated checkpoint was verified, so weights availability remains uncertain. Checked 2026-10-10 · medium confidence Runner H H-LLM 2BUncertain weights · Tiny · Published size · 2B total H Company’s historical report specifies a 2B language runtime model. This is the original internal H-LLM, not a later Holo checkpoint. No downloadable evaluated checkpoint was verified, so weights availability remains uncertain. Checked 2026-10-10 · medium confidence Runner H 0.1 external model stack (undisclosed)Uncertain weights · Large · Estimated The author says Runner H combines internal and external models but does not identify the external runtime models. Large is a provisional frontier-provider prior, not a disclosed parameter or price measurement. GPT-4o is named as a scoring evaluator only and does not establish this runtime identity. Checked 2026-10-10 · low confidence | 67% | Uncertain weightsLargeEstimatedLargest component | 2025-03 | Source ↗ |
| 19 | WebVoyagerAcademic Research · HarnessEvaluation & model detailsBaseline implementation using standard multimodal LLM control. Original agent paper uses gpt-4-vision-preview. GPT-4 Turbo used to generate tasks and the external evaluation judge are excluded from runtime classification. Public Selenium agent code is separate from the closed model. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-4 Vision preview; parameters are undisclosed. GPT-4 Vision previewClosed weights · Large · Price proxy Token price: $10 input / $30 output per million. Blended 3:1: $15/M. Original GPT-4 Vision preview, also listed as gpt-4-1106-vision-preview in OpenAI deprecations. Parameters undisclosed. Historical standard uncached $10/$30 token rates give a $15 blend; model was retired in December 2024.
Checked 2026-10-10 · high confidence | 59.1% | Closed weightsLargePrice proxy | 2024-01 | Source ↗Repository ↗ |
| 20 | Anthropic Computer Use 3.5Anthropic · HarnessEvaluation & model detailsSampled 50/602 tasks for direct comparison; reported on trykura.com. Kura’s comparison uses Anthropic’s official computer-use demo. October 2024 upgraded Sonnet is the supported historical computer-use model; this is a release-date inference rather than a snapshot log. The 56% comparison is on 50 tasks. Demo code availability does not imply open weights. Harness code: open. Price proxy: the size class comes from the standard token price of Claude 3.5 Sonnet (20241022); parameters are undisclosed. Claude 3.5 Sonnet (20241022)Closed weights · Large · Price proxy Token price: $3 input / $15 output per million. Blended 3:1: $6/M. October 2024 upgraded Sonnet introduced computer use. Parameters undisclosed. Anthropic kept the predecessor’s standard $3/$15 token prices, giving a $6 blend. Historical Kura comparison identifies the official computer-use demo, not an explicitly logged snapshot. Checked 2026-10-10 · high confidence | 56.0% | Closed weightsLargePrice proxy | 2024-11 | Source ↗ |
| 21 | WILBURBardeen / UC Berkeley · HarnessEvaluation & model detailsResearch implementation using black-box optimization techniques. Paper pins GPT-4-0125-preview and includes text-embedding-3-large retrieval plus a learned runtime MLP ranker. No trained ranker download or full public harness was verified on the author project. External GPT-4V scoring and autocurriculum-only calls are excluded. Harness code: uncertain. Price proxy: the size class comes from the standard token price of GPT-4 Turbo (0125 preview); parameters are undisclosed. Size reflects the largest of 3 models; parameters are not summed. GPT-4 Turbo (0125 preview)Closed weights · Large · Price proxy Token price: $10 input / $30 output per million. Blended 3:1: $15/M. Exact January 2024 preview used by WILBUR. Parameters undisclosed; published standard uncached $10/$30 token rates give a $15 blend. Not the later stable GPT-4 Turbo snapshot.
Checked 2026-10-10 · high confidence text-embedding-3-largeClosed weights · Small · Estimated Hosted embedding helper with undisclosed parameters. Published input price is $0.13 per 1M tokens; there is no generated-output tariff, so the required 3:1 text-generation proxy cannot be calculated. Provisional Small follows its low input cost, not the word large in its name. Checked 2026-10-10 · low confidence WILBUR demonstration rankerUncertain weights · Tiny · Published size · 0.002B total Runtime demonstration-ranking MLP. The published 7680→200→200→1 fully connected architecture implies 1,576,601 parameters including biases; this is an architecture-derived count, not an inspected checkpoint. The author project links the paper and hosted product but no trained ranker download was verified. Checked 2026-10-10 · medium confidence | 53% | Uncertain weightsLargePrice proxyLargest component | 2024-04 | Source ↗ |
| 22 | GPT-4 (All Tools)OpenAI · SystemEvaluation & model detailsChatGPT integrated tool baseline from original WebVoyager paper; reported on arxiv.org. Paper’s ChatGPT GPT-4 All Tools product baseline has no pinned API checkpoint or complete tool-runtime disclosure. Retain an explicitly provisional historical system identity instead of assigning a later GPT-4 Turbo model. Harness code: closed. Estimated for ChatGPT GPT-4 All Tools (historical baseline): no verified parameter count or standard token price, so the size class is provisional. ChatGPT GPT-4 All Tools (historical baseline)Closed weights · Large · Estimated The WebVoyager paper evaluates the GPT-4 All Tools ChatGPT product, not an identified API snapshot. Its browser/tool stack and exact underlying model are unpinned. Provisional Large follows the historical GPT-4 product family; subscription cost is not substituted for a token price. Checked 2026-10-10 · low confidence | 30.8% | Closed weightsLargeEstimated | 2024-01 | Source ↗ |
About this benchmark
WebVoyager evaluates end-to-end browser agents on 643 tasks across 15 popular real-world websites. Tasks cover search, navigation, form filling, map and travel lookup, shopping, and information retrieval on live pages rather than static snapshots.
It is useful as a browser-agent adoption signal because many commercial and open-source agents report it, but it is unusually sensitive to task drift, removed tasks, evaluator choice, and whether the run used the full task suite.
Read each row as a full-system result: model, prompt, browser execution layer, retry policy, DOM or accessibility extraction, and visual grounding can all contribute to the final score.
WebVoyager is high-visibility but not fully standardized across modern submissions; small score gaps can reflect setup choices as much as capability.
Example tasks
Three public tasks quoted from benchmark sources:
- "Provide a recipe for vegetarian lasagna with more than 100 reviews and a rating of at least 4.5 stars suitable for 6 people." Citation: WebVoyager dataset
- "Search an Xbox Wireless controller with green color and rated above 4 stars." Citation: WebVoyager dataset
- "Find a Blue iPhone 12 Pro 128gb and add to cart." Citation: WebVoyager dataset
Methodology
- Primary metric is task success rate: completed tasks divided by evaluated tasks. The original paper used GPT-4V as an automatic evaluator and reported 85.3% agreement with human judgment.
- We prioritize public sources that identify the system, score, task subset or evaluator when available, and a paper, repository, model card, or launch post that can be checked later.
- Direct comparisons are strongest when systems run the same task set, same evaluator, same attempt policy, and same handling of stale or auth-gated tasks.
- Rows that use filtered task subsets, manual correction, or custom judges are kept when source-linked, but notes should be read before treating adjacent ranks as meaningful differences.
Links
Related benchmarks
Compare this benchmark with related pages from the hub: