Scores
How model sizes and weights are classified
Size classes follow the Artificial Analysis categories: Tiny ≤4B; Small >4–40B; Medium >40–150B; Large >150B total parameters. Active MoE parameters are reported separately.
When parameters are unavailable, we use standard uncached token prices with a 3:1 input/output blend: Small <$0.15/M, Medium $0.15–$1/M, Large >$1/M. “Price proxy” is a pricing category, not a parameter estimate. “Estimated” marks a provisional classification and remains included in filters. “Unclassified” results await model research and match only the all and mixed / uncertain weight filters.
Harnesses inherit the evaluated model stack's class. Multi-model systems use the largest component category as an approximation. Open weights requires released weights for every required model; open harness code alone does not qualify. Unverified backbones and unavailable evaluated checkpoints are marked uncertain.
Results compare evaluated systems, including their tools, compute budgets, and context strategies. Size categories do not guarantee hardware compatibility. Suggest a correction ↗
Results
Agent scope · 52 systems| Rank | System / submission | Score ↓ | Model class | Reported | Source |
|---|---|---|---|---|---|
| 1 | WebTactix (DeepSeek v3.2)WebTactix · SystemEvaluation & model detailsProject reports 594 correct, 205 failed, and 13 N/A among 812 tasks; 74.34% excludes the 13 N/A tasks (594/799). DeepSeek v3.2 runtime. WebTactix semantic-tree search uses the reported DeepSeek-V3.2 backbone across constraint, extraction, planning and decision roles. Public code shares one selected LLM client; the current deepseek-chat alias is mutable, so the historical named version is retained. Harness code: open. Published size: from the parameter count of DeepSeek-V3.2. DeepSeek-V3.2Open weights · Large · Published size · 685.355B total MIT downloadable weights verified. Total is publisher safetensors metadata, rounded in billions, including auxiliary tensors. V3.2 adds sparse-attention indexing to the V3.1 architecture; active count was not independently established here. Current API pricing page now describes successor models, so historical V3.2 prices remain unset.
Checked 2026-10-10 · high confidence | 74.3% | Open weightsLarge | 2026-02 | Source ↗Repository ↗ |
| 2 | OpAgentCodeFuse AI · SystemEvaluation & model detailsOpen-source multi-agent browser system; repository publishes the WebArena result and final trajectories. Paper section 4.3 uses Gemini-3-Pro for planner, reflector and summarizer, plus a separately post-trained Qwen2.5-VL-72B-MFT grounder. Released OpAgent-32B belongs to the later single-model mode and cannot identify this 71.6 system. Harness code: open. Estimated for Gemini 3 Pro: no verified parameter count or standard token price, so the size class is provisional. Size reflects the largest of 2 models; parameters are not summed. Gemini 3 ProClosed weights · Large · Estimated Provisional Large based on Google’s flagship Pro positioning. Parameters are undisclosed; current Google pricing pages have replaced this version with Gemini 3.1 Pro, so no exact-version current token rate is asserted.
Checked 2026-10-10 · low confidence OpAgent Qwen2.5-VL-72B-MFT grounderUncertain weights · Medium · Published size · 72B total Paper section 4.3 names this post-trained 72B grounder. The public organization inventory lists OpAgent-32B, a different later single-model release; no matching 72B-MFT checkpoint was found. Open base weights and public orchestration code do not establish release of this fine-tune. Checked 2026-10-10 · medium confidence | 71.6% | Uncertain weightsLargeEstimatedLargest component | 2026-01 | Source ↗Repository ↗ |
| 3 | ColorBrowserAgentMadeAgents · HarnessEvaluation & model detailsHuman-like browser agent result; code and trajectories are public in the MadeAgents repository. ColorBrowserAgent paper and launch README explicitly use GPT-5. Site-prior generation belongs to the same framework; no distinct required runtime model is identified. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-5; parameters are undisclosed. GPT-5Closed weights · Large · Price proxy Token price: $1.25 input / $10 output per million. Blended 3:1: $3.4375/M. Parameter count is undisclosed. Large is a price proxy from standard uncached API text rates; it does not assert a physical parameter size. Tool fees and cached, batch, flex, and priority rates are excluded. Checked 2026-10-10 · high confidence | 71.2% | Closed weightsLargePrice proxy | 2025-12 | Source ↗Repository ↗ |
| 4 | Claude Code + GBOX MCPGBOX AI · HarnessEvaluation & model detailsClaude Code backed by GBOX MCP scaffolding; public report and trajectories are in the BabelCloud WebArena repository. Published agent defaults and an October 25 execution log identify Bedrock claude-sonnet-4-5-20250929. Logs and the report also require GBOX semantic visual targeting; a separate hosted coordinate model is represented explicitly because its historical identity and weights are unverified. Current gbox-handy-1 documentation is not an exact historical model pin. Public orchestration code is separate from the hosted grounding service. Harness code: open. Estimated for GBOX historical visual-coordinate model (unidentified): no verified parameter count or standard token price, so the size class is provisional. Size reflects the largest of 2 models; parameters are not summed.
Claude Sonnet 4.5Closed weights · Large · Price proxy Token price: $3 input / $15 output per million. Blended 3:1: $6/M. Closed hosted model with undisclosed parameter count. Published standard uncached first-party text prices are $3 input/$15 output per million tokens, giving a $6/M blend and Large price-proxy category. Cache, batch, and tool charges are excluded. This uses the standard context tier (up to 200K input tokens), not long-context premium pricing. Checked 2026-10-10 · high confidence GBOX historical visual-coordinate model (unidentified)Uncertain weights · Large · Estimated The reported October 2025 runtime uses natural-language screenshot targeting delegated to GBOX, in addition to Claude reasoning. Current documentation names gbox-handy-1, but does not pin the historical run or disclose parameters, token prices, or matching released weights. Large is a provisional estimate allowing an undisclosed frontier vision backend; no numerical parameter claim is made. The known Sonnet component already places the complete stack in Large.
Checked 2026-10-11 · low confidence | 68.0% | Uncertain weightsLargeEstimatedLargest component | 2025-10-25 | Source ↗Repository ↗ |
| 5 | DeepSky AgentDeepSky · SystemEvaluation & model detailsPublic tracker row; no separate public source or repository link was found in the tracker. The tracker does not disclose a backbone or implementation link. Full model stack remains a labeled estimate. Harness code: uncertain. Estimated for DeepSky Agent historical model stack: no verified parameter count or standard token price, so the size class is provisional. DeepSky Agent historical model stackUncertain weights · Large · Estimated Only the public tracker corroborates this named system; no primary model identity, released checkpoint or token tariff was found. Large is a low-confidence estimate for the high-performing managed agent, not a claimed model size. Checked 2026-10-10 · low confidence | 66.9% | Uncertain weightsLargeEstimated | 2025-09 | Source ↗ |
| 6 | Narada AINarada AI · SystemEvaluation & model detailsNarada blog and public tracker report the WebArena result; no public code repository was found. Vendor-operated system. The blog establishes the result and design lineage without identifying exact inference models. Harness code: closed. Estimated for Narada Operator historical model stack: no verified parameter count or standard token price, so the size class is provisional. Narada Operator historical model stackUncertain weights · Large · Estimated The vendor post describes a proprietary operator and links Plan-and-Act research, but does not name the exact models behind 64.16. Large is a provisional frontier-system estimate; research backbones are not automatically the deployed stack. Checked 2026-10-10 · low confidence | 64.2% | Uncertain weightsLargeEstimated | 2025-10 | Source ↗ |
| 7 | IBM CUGAIBM · SystemEvaluation & model detailsCUGA reports 61.7% on WebArena and publishes code, evaluation artifacts, and results. CUGA orchestration is public, but the exact historical WebArena inference stack is not disclosed. The later AppWorld GPT-4.1 table is a different benchmark. Harness code: open. Estimated for IBM CUGA historical WebArena model stack: no verified parameter count or standard token price, so the size class is provisional. IBM CUGA historical WebArena model stackUncertain weights · Large · Estimated The historical paper says its SOTA system uses a frontier model and that smaller/open-model experiments are future work, but does not pin the 61.7 WebArena stack. Large is provisional. GPT-4.1 in a later AppWorld table and modern gpt-4o configuration examples are not substituted. Checked 2026-10-10 · low confidence | 61.7% | Uncertain weightsLargeEstimated | 2025-02-17 | Source ↗Repository ↗ |
| 8 | Kimi K2.5Moonshot AI · ModelEvaluation & model detailsTechnical report reports GUI-based browsing without external tools; Kimi K2.5 weights are publicly released. Technical report identifies GUI-only Kimi K2.5 browsing without external tools. Weights are released; exact benchmark harness release was not verified. This is not Agent Swarm. Harness code: uncertain. Published size: from the parameter count of Kimi K2.5. Kimi K2.5Open weights · Large · Published size · 1,000B total · 32B active Modified-MIT weights verified in public repository (64 safetensors shards). Swarm orchestration is distinct from these model weights. Price table was absent from retrieved pricing text and direct retrieval was blocked; token rates omitted. Checked 2026-10-10 · high confidence | 58.9% | Open weightsLarge | 2026-02 | Source ↗Repository ↗ |
| 9 | OpenAI OperatorOpenAI · SystemEvaluation & model detailsOpenAI Computer-Using Agent result reported with the Operator system card and public WebArena tracker. Original January 2025 Operator/CUA result. Later computer-use-preview API pricing does not establish the original evaluated model's price. Harness code: closed. Estimated for OpenAI original Computer-Using Agent (January 2025): no verified parameter count or standard token price, so the size class is provisional. OpenAI original Computer-Using Agent (January 2025)Closed weights · Large · Estimated The original Operator CUA combines GPT-4o vision with reinforcement-learned reasoning. Its parameter count and January evaluation token prices are undisclosed. Large is a provisional category informed by the GPT-4o-based frontier architecture and later computer-use-preview pricing, not a parameter estimate or assertion that the March API snapshot is identical. Do not substitute later o3-based Operator or computer-use-preview-2025-03-11 for this historical model.
Checked 2026-10-10 · medium confidence | 58.1% | Closed weightsLargeEstimated | 2025-01 | Source ↗ |
| 10 | Jace.AI (AWA-1.5)Jace AI · SystemEvaluation & model detailsJace reports AWA 1.5 at 57.14% on WebArena; implementation is proprietary. Proprietary AWA-1.5 system; underlying checkpoints remain undisclosed. Harness code: closed. Estimated for Jace AWA-1.5 model stack: no verified parameter count or standard token price, so the size class is provisional. Jace AWA-1.5 model stackUncertain weights · Large · Estimated The vendor report names AWA-1.5 and proprietary DOM processing but does not disclose participating checkpoints, their parameters, or standard token rates. Large is a provisional managed-agent estimate; weight availability remains unresolved. Checked 2026-10-10 · low confidence | 57.1% | Uncertain weightsLargeEstimated | 2024-08 | Source ↗ |
| 11 | WebChallenger (GLM-4-32B)NewIndependent Research · HarnessEvaluation & model detailsWebChallenger paper reports 56.3% over all 812 WebArena tasks using GLM-4-32B combined with Qwen-2.5-VL-7B as a vision model. Paper explicitly uses GLM-4-32B-0414 controller plus Qwen2.5-VL-7B-Instruct image captioning on WebArena. Qwen3-VL-4B is for VisualWebArena and is excluded here. Harness code: open. Published size: from the parameter count of GLM-4-32B-0414, Qwen2.5-VL-7B-Instruct. Size reflects the largest of 2 models; parameters are not summed. GLM-4-32B-0414Open weights · Small · Published size · 32B total Publisher nominal 32B dense architecture; the exact off-the-shelf controller used by WebChallenger. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence Qwen2.5-VL-7B-InstructOpen weights · Small · Published size · 8.292B total Publisher repository tensor total rounded in billions includes the visual component; the nominal 7B name refers to the model scale. This remains Small. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 56.3% | Open weightsSmallLargest component | 2026-06-09 | Source ↗Repository ↗ |
| 12 | Plan-MCTS + GPT-5-miniAcademic Research · HarnessEvaluation & model detailsPlan-MCTS paper reports 55.3% on the full 812-task WebArena test set with GPT-5-mini. Table 2 identifies GPT-5-mini for the plan-search configuration. Planning, operating and dual-gating are inference roles of the selected backbone. No released implementation link was found in the retrieved paper. Harness code: uncertain. Price proxy: the size class comes from the standard token price of GPT-5 mini; parameters are undisclosed. GPT-5 miniClosed weights · Medium · Price proxy Token price: $0.25 input / $2 output per million. Blended 3:1: $0.6875/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. Published blend is $0.6875/M: Medium despite the mini product name. Checked 2026-10-10 · high confidence | 55.3% | Closed weightsMediumPrice proxy | 2026-02 | Source ↗ |
| 13 | WebOperator + GPT-4oKAIST KAG NLP · SystemEvaluation & model detailsWebOperator paper and project report 54.6% with GPT-4o; code and experiment artifacts are public. The paper calls its backend gpt-4o-2025-01-01, which is not a documented OpenAI snapshot. Released configuration uses an Azure deployment alias gpt-4o for action, reward and checklist generation, plus all-MiniLM-L6-v2 for semantic action deduplication. Preserve exact-version uncertainty. Harness code: open. Estimated for GPT-4o (historical snapshot unspecified): no verified parameter count or standard token price, so the size class is provisional. Size reflects the largest of 2 models; parameters are not summed. GPT-4o (historical snapshot unspecified)Closed weights · Large · Estimated Reports/configurations name GPT-4o without a verifiable snapshot or use an unpinned Azure deployment alias. Keep separate from May, August and November 2024 checkpoints. Large is provisional from known GPT-4o standard price tiers; no exact tariff or parameter count is assigned to these historical runs.
Checked 2026-10-10 · low confidence all-MiniLM-L6-v2Open weights · Tiny · Published size · 0.023B total Repository tensor total. WebOperator enables this sentence embedding model to deduplicate candidate actions at runtime. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 54.6% | Mixed weightsLargeEstimatedLargest component | 2025-12 | Source ↗Repository ↗ |
| 14 | ScribeAgent + GPT-4oAcademic Research · SystemEvaluation & model detailsScribeAgent reports 53.0% with GPT-4o; the tracker marks data as proprietary. The 53.0 result combines the 32B ScribeAgent-Large fine-tune with GPT-4o for task decomposition/action-representation mapping. Proprietary training data does not prove whether exact fine-tuned weights were released; no matching release was located. Harness code: uncertain. Estimated for GPT-4o (historical snapshot unspecified): no verified parameter count or standard token price, so the size class is provisional. Size reflects the largest of 2 models; parameters are not summed. ScribeAgent-Large 32BUncertain weights · Small · Published size · 32B total The paper identifies a Qwen2.5-32B fine-tune on proprietary workflow data. No evaluated checkpoint was linked in the inspected paper or found by Hugging Face search; do not infer open weights from the base. Checked 2026-10-10 · medium confidence GPT-4o (historical snapshot unspecified)Closed weights · Large · Estimated Reports/configurations name GPT-4o without a verifiable snapshot or use an unpinned Azure deployment alias. Keep separate from May, August and November 2024 checkpoints. Large is provisional from known GPT-4o standard price tiers; no exact tariff or parameter count is assigned to these historical runs.
Checked 2026-10-10 · low confidence | 53.0% | Uncertain weightsLargeEstimatedLargest component | 2024-12-24 | Source ↗ |
| 15 | AgentSymbioticAcademic Research · SystemEvaluation & model detailsAgentSymbiotic reports 52.1%; an anonymized public code artifact is linked from the tracker. Table 1 assigns 52.1 to the Claude-3.5 RAG agent. Runtime trajectory embeddings are unidentified. Small distilled models and multi-LLM data synthesis are training machinery; optional privacy hybrid experiments do not establish they ran in this score configuration. Anonymous code retrieval returned HTTP 403. Harness code: uncertain. Estimated for Claude 3.5 Sonnet (historical unpinned snapshot): no verified parameter count or standard token price, so the size class is provisional. Size reflects the largest of 2 models; parameters are not summed. Claude 3.5 Sonnet (historical unpinned snapshot)Closed weights · Large · Estimated The source identifies Claude 3.5 Sonnet but not its June/October snapshot. Provisional Large follows this proprietary frontier family; parameters and an exact evaluated-version tariff are not asserted. Snapshot-specific records remain separate. Checked 2026-10-10 · low confidence AgentSymbiotic undisclosed retrieval helperUncertain weights · Small · Estimated Runtime trajectory similarity search requires embeddings; the retrieved paper does not identify the embedding checkpoint, and the linked anonymous code endpoint returned HTTP 403. Small is a provisional category for this specialized retrieval component, not a parameter assertion. Main Claude reasoning determines the overall Large category. Checked 2026-10-10 · low confidence | 52.1% | Uncertain weightsLargeEstimatedLargest component | 2025-01 | Source ↗Repository ↗ |
| 16 | Nemotron 3.5 Bolt 74B-A9B-24GBNewNVIDIA · ModelEvaluation & model detailsIQ2-quantized (24GB) release of the open-weight 74B-A9B model; full 812-task set with a 100-step limit. Self-reported on the NVIDIA model card. IQ2-quantized 24GB checkpoint on the full 812-task set with a 100-step limit; self-reported on the NVIDIA model card. Harness code: uncertain. Published size: from the parameter count of Nemotron 3.5 Bolt 74B-A9B-24GB. Nemotron 3.5 Bolt 74B-A9B-24GBOpen weights · Medium · Published size · 74B total · 9B active NVIDIA publishes 74B total / 9B active parameters. IQ2-quantized checkpoint, about 24GB. Open weights under the OpenMDW License Agreement v1.1. Checked 2026-10-14 · high confidence | 50.86% | Open weightsMedium | 2026-10-15 | Source ↗Repository ↗ |
| 17 | Nemotron 3.5 Bolt 74B-A9B-BF16NewNVIDIA · ModelEvaluation & model detailsFull-precision BF16 weights of the open-weight 74B-A9B model; full 812-task set with a 100-step limit. Self-reported on the NVIDIA model card. BF16 checkpoint on the full 812-task set with a 100-step limit; self-reported on the NVIDIA model card. Harness code: uncertain. Published size: from the parameter count of Nemotron 3.5 Bolt 74B-A9B-BF16. Nemotron 3.5 Bolt 74B-A9B-BF16Open weights · Medium · Published size · 74B total · 9B active NVIDIA publishes 74B total / 9B active parameters. Full-precision BF16 checkpoint. Open weights under the OpenMDW License Agreement v1.1. Checked 2026-10-14 · high confidence | 50.25% | Open weightsMedium | 2026-10-15 | Source ↗Repository ↗ |
| 18 | Learn-by-InteractAcademic Research · SystemEvaluation & model detailsPaper reports 48.0% on WebArena; no public implementation repository was found. The 48.0 result is Claude-3.5 Sonnet with in-context agentic retrieval, not the 22B fine-tune. Vertex text embeddings perform runtime dense retrieval. Gemini filtering and Claude grading used for dataset/evaluation preparation are excluded. Harness code: uncertain. Estimated for Claude 3.5 Sonnet (historical unpinned snapshot): no verified parameter count or standard token price, so the size class is provisional. Size reflects the largest of 2 models; parameters are not summed. Claude 3.5 Sonnet (historical unpinned snapshot)Closed weights · Large · Estimated The source identifies Claude 3.5 Sonnet but not its June/October snapshot. Provisional Large follows this proprietary frontier family; parameters and an exact evaluated-version tariff are not asserted. Snapshot-specific records remain separate. Checked 2026-10-10 · low confidence Learn-by-Interact historical Vertex embedding helperUncertain weights · Small · Estimated The appendix names Vertex AI text embeddings for runtime dense retrieval but gives no model version. The mutable linked guide now demonstrates newer Gemini embeddings, which cannot identify the January 2025 helper. Small is a low-confidence specialized-retrieval estimate; no count or output-token price is fabricated. Checked 2026-10-10 · low confidence | 48.0% | Uncertain weightsLargeEstimatedLargest component | 2025-01 | Source ↗ |
| 19 | WebUncertainty + GPT-4-TurboNewAcademic Research · HarnessEvaluation & model detailsWebUncertainty reports 46.9% with GPT-4-Turbo on WebArena and publishes the project repository. Paper explicitly pins GPT-4-turbo-2024-04-09. Public repository currently contains README, project assets and dependency metadata but no executable agent implementation; a public repository alone does not establish released harness code. Harness code: uncertain. Price proxy: the size class comes from the standard token price of GPT-4 Turbo (2024-04-09); parameters are undisclosed. GPT-4 Turbo (2024-04-09)Closed weights · Large · Price proxy Token price: $10 input / $30 output per million. Blended 3:1: $15/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. The stable Turbo snapshot is distinct from earlier previews. Checked 2026-10-10 · high confidence | 46.9% | Closed weightsLargePrice proxy | 2026-04 | Source ↗Repository ↗ |
| 20 | AgentOccam-JudgeAmazon Science · HarnessEvaluation & model detailsAgentOccam-Judge public tracker row; Amazon Science publishes the AgentOccam implementation. AgentOccam uses GPT-4-Turbo-2024-04-09; the added runtime action-selection judge is GPT-4-Turbo as well. This judge affects actions and is included through the same deduplicated model ID, unlike a post-run scoring judge. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-4 Turbo (2024-04-09); parameters are undisclosed. GPT-4 Turbo (2024-04-09)Closed weights · Large · Price proxy Token price: $10 input / $30 output per million. Blended 3:1: $15/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. The stable Turbo snapshot is distinct from earlier previews. Checked 2026-10-10 · high confidence | 45.7% | Closed weightsLargePrice proxy | 2024-10 | Source ↗Repository ↗ |
| 21 | A3-Qwen3.5-9BNewMcGill NLP · ModelEvaluation & model detailsBrowserGym leaderboard discussion reports 42.1%; the paper/model card table reports 41.5 under its project setup. Exact A3 fine-tune has released weights and evaluation configuration. Stored 42.1 submission differs from the model card's 41.5 project setup; scores are preserved. Harness code: open. Published size: from the parameter count of A3-Qwen3.5-9B. A3-Qwen3.5-9BOpen weights · Small · Published size · 9B total Released 9B multimodal fine-tune. Model card reports 41.5 while the BrowserGym submission reports 42.1; result data is preserved. Training annotator Gemini is excluded. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 42.1% | Open weightsSmall | 2026-04 | Source ↗Repository ↗ |
| 22 | GUI-API Hybrid AgentAcademic Research · HarnessEvaluation & model detailsLatest arXiv version reports 38.9%; the public tracker still lists 35.8 for an earlier or alternate setup. Paper specifies GPT-4o for the browsing/API/hybrid agent, without a snapshot. APIs use two-stage documentation access; no additional named learned inference component is established. OpenHands is the framework, but release of this exact extension was not verified. Harness code: uncertain. Estimated for GPT-4o (historical snapshot unspecified): no verified parameter count or standard token price, so the size class is provisional. GPT-4o (historical snapshot unspecified)Closed weights · Large · Estimated Reports/configurations name GPT-4o without a verifiable snapshot or use an unpinned Azure deployment alias. Keep separate from May, August and November 2024 checkpoints. Large is provisional from known GPT-4o standard price tiers; no exact tariff or parameter count is assigned to these historical runs.
Checked 2026-10-10 · low confidence | 38.9% | Closed weightsLargeEstimated | 2025-06 | Source ↗ |
| 23 | WebPilotAcademic Research · HarnessEvaluation & model detailsPaper reports 37.2%; the public tracker notes no open-source code or trajectories were released. Paper footnote pins GPT-4o-2024-05-13 for the reported WebPilot configuration. GPT-3.5-0125 is an alternative experiment, not an additional required model. No matching released implementation was identified. Harness code: uncertain. Price proxy: the size class comes from the standard token price of GPT-4o (2024-05-13); parameters are undisclosed. GPT-4o (2024-05-13)Closed weights · Large · Price proxy Token price: $5 input / $15 output per million. Blended 3:1: $7.5/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. Legacy standard table explicitly prices this May snapshot at $5/$15. Do not apply the later GPT-4o alias's $2.50/$10 rates. Checked 2026-10-10 · high confidence | 37.2% | Closed weightsLargePrice proxy | 2024-08 | Source ↗ |
| 24 | Agent Workflow MemoryAcademic Research · HarnessEvaluation & model detailsWorkflow-memory agent result; public repository is available with the paper artifacts. The paper pins GPT-4-0613 for action generation and workflow induction. Its online success predictor gates which experiences become memory for later test tasks, so it belongs to the runtime stack. Historical public code has conflicting GPT-3.5-Turbo/GPT-4o defaults and does not establish the evaluator used for the reported 35.5; that helper remains explicitly unpinned. Harness code: open. Estimated for AWM memory evaluator (OpenAI model unpinned): no verified parameter count or standard token price, so the size class is provisional. Size reflects the largest of 2 models; parameters are not summed.
GPT-4 (0613)Closed weights · Large · Price proxy Token price: $30 input / $60 output per million. Blended 3:1: $37.5/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. Official default GPT-4 snapshot is 0613; historical model, not GPT-4 Turbo. Checked 2026-10-10 · high confidence AWM memory evaluator (OpenAI model unpinned)Closed weights · Large · Estimated Online AWM uses neural success predictions to select workflows retained for later test tasks, making the evaluator a runtime component. Historical code defaults conflict: trajectory evaluation names GPT-3.5-Turbo while induction expects a GPT-4o-named evaluation file; the exact published-run configuration is unavailable. Large is a provisional category covering the possible GPT-4/GPT-4o component, not a proven tariff or parameter count. All documented candidates are closed OpenAI models.
Checked 2026-10-10 · low confidence | 35.5% | Closed weightsLargeEstimatedLargest component | 2024-09 | Source ↗Repository ↗ |
| 25 | StePASAPP Research · HarnessEvaluation & model detailsPublic tracker reports 33.5%; the SteP project publishes code and WebArena materials. Paper specifies the GPT-4-Turbo-Preview alias. Repository now documents newer stable April Turbo runs; those runs do not pin the original 33.5 result. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-4 Turbo Preview (historical unpinned alias); parameters are undisclosed. GPT-4 Turbo Preview (historical unpinned alias)Closed weights · Large · Price proxy Token price: $10 input / $30 output per million. Blended 3:1: $15/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. The SteP paper names gpt-4-turbo-preview without a date; the preview tariff is published, but the exact evaluation snapshot is unpinned. Later repository runs use stable April Turbo and must not replace the paper result. Checked 2026-10-10 · medium confidence | 33.5% | Closed weightsLargePrice proxy | 2024-04 | Source ↗Repository ↗ |
| 26 | TTIAcademic Research · ModelEvaluation & model detailsTest-time interaction result; WebArena artifacts are public on Hugging Face. Released WebArena Gemma3-12B checkpoint corresponds to TTI. Training uses a Gemma3-27B evaluator, which is excluded from the inference stack. Exact public runtime code availability was not established. Harness code: uncertain. Published size: from the parameter count of TTI WebArena Gemma 3 12B. TTI WebArena Gemma 3 12BOpen weights · Small · Published size · 12B total The paper identifies the 12B base and the released config matches Gemma3ForConditionalGeneration. The full pytorch_model.bin checkpoint is listed. Its Gemma 3 27B rollout evaluator is for training, not inference. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 26.1% | Open weightsSmall | 2025-06 | Source ↗Repository ↗ |
| 27 | BrowserGym + GPT-4oServiceNow Research · HarnessEvaluation & model detailsWorkArena/BrowserGym reports 23.5% for GPT-4o-2024-05-13 on 812 WebArena tasks; open BrowserGym harness. The WorkArena/BrowserGym paper explicitly assigns the 23.5% WebArena result to gpt-4o-2024-05-13. The public label was corrected from GPT-4 to GPT-4o; score is preserved. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-4o (2024-05-13); parameters are undisclosed. GPT-4o (2024-05-13)Closed weights · Large · Price proxy Token price: $5 input / $15 output per million. Blended 3:1: $7.5/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. Legacy standard table explicitly prices this May snapshot at $5/$15. Do not apply the later GPT-4o alias's $2.50/$10 rates. Checked 2026-10-10 · high confidence | 23.5% | Closed weightsLargePrice proxy | 2024-04 | Source ↗Repository ↗ |
| 28 | AgentTrek-1.0-32BxLang AI · ModelEvaluation & model detailsAgentTrek project reports 22.4%; code, model, and data links are public. Released AgentTrek-1.0-32B fine-tune and evaluation scripts are linked by the official repository. Training/data-generation teachers are excluded. Harness code: open. Published size: from the parameter count of AgentTrek-1.0-32B. AgentTrek-1.0-32BOpen weights · Small · Published size · 32B total Fine-tuned Qwen2.5-32B-Instruct. Data-generation and reward models are not runtime dependencies. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 22.4% | Open weightsSmall | 2025-01 | Source ↗Repository ↗ |
| 29 | GPT-4 + Auto EvalAcademic Research · SystemEvaluation & model detailsAutomatic evaluator study reports a GPT-4 WebArena result; no separate implementation repository was found. Public reproduction script uses GPT-4-1106-preview for the actor/reflection loop and GPT-4V (1106 vision preview in the paper) for runtime success feedback that triggers retries. The visual evaluator is a required runtime component here; captioner/Mixtral alternatives are separate experiments. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-4 Turbo (1106 preview), GPT-4 Vision preview; parameters are undisclosed. Size reflects the largest of 2 models; parameters are not summed. GPT-4 Turbo (1106 preview)Closed weights · Large · Price proxy Token price: $10 input / $30 output per million. Blended 3:1: $15/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. November 2023 text preview. The evaluated alias is pinned in the AutoEval runtime script; the provider retains the GPT-4 Turbo Preview tariff. Exact snapshot page returned 404, while the preview family documentation and deprecation listing remain accessible. Checked 2026-10-10 · medium confidence GPT-4 Vision previewClosed weights · Large · Price proxy Token price: $10 input / $30 output per million. Blended 3:1: $15/M. Original GPT-4 Vision preview, also listed as gpt-4-1106-vision-preview in OpenAI deprecations. Parameters undisclosed. Historical standard uncached $10/$30 token rates give a $15 blend; model was retired in December 2024.
Checked 2026-10-10 · high confidence | 20.2% | Closed weightsLargePrice proxyLargest component | 2024-04 | Source ↗ |
| 30 | GPT-4o + Tree SearchAcademic Research · HarnessEvaluation & model detailsTree-search agent result from the Search Agents project page. The search-agents paper pins GPT-4o-2024-05-13. Policy and value roles use GPT-4o; the WebArena text-only setting does not use BLIP-2, which belongs to caption-augmented VisualWebArena experiments. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-4o (2024-05-13); parameters are undisclosed. GPT-4o (2024-05-13)Closed weights · Large · Price proxy Token price: $5 input / $15 output per million. Blended 3:1: $7.5/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. Legacy standard table explicitly prices this May snapshot at $5/$15. Do not apply the later GPT-4o alias's $2.50/$10 rates. Checked 2026-10-10 · high confidence | 19.2% | Closed weightsLargePrice proxy | 2024-06 | Source ↗ |
| 31 | AutoWebGLMTHUDM · ModelEvaluation & model detailsAutoWebGLM paper reports 18.2%; official repository includes WebArena evaluation materials. Official repository identifies ChatGLM3-6B as the trained backbone and publishes evaluation code. No matching fine-tuned checkpoint was verified; GPT data-generation teachers are excluded. Harness code: open. Published size: from the parameter count of AutoWebGLM 6B. AutoWebGLM 6BUncertain weights · Small · Published size · 6B total Official repository identifies a trained ChatGLM3-6B agent. It links inference instructions for the base but no matching downloadable fine-tuned checkpoint; Hugging Face model search returned no AutoWebGLM result. Public harness code is separate from weight availability. Checked 2026-10-10 · medium confidence | 18.2% | Uncertain weightsSmall | 2024-04 | Source ↗Repository ↗ |
| 32 | NNetNavStanford NLP · ModelEvaluation & model detailsNNetNav result with public model checkpoint linked from the tracker. Use the WebArena-specific released Llama-3.1-8B checkpoint. The larger 70B exploration/reward model is used to construct training data, not the evaluated student's runtime. Harness code: open. Published size: from the parameter count of NNetNav Llama-3.1-8B WebArena. NNetNav Llama-3.1-8B WebArenaOpen weights · Small · Published size · 8B total Released WebArena-specific checkpoint; the card directly matches the stored 16.3 result. Llama-3.1-70B generated/filtered training trajectories and is excluded from inference. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 16.3% | Open weightsSmall | 2025-01 | Source ↗Repository ↗ |
| 33 | gpt-4-0613 (no not-achievable hint)OpenAI · ModelEvaluation & model detailsOriginal WebArena baseline without providing the not-achievable task hint. Original exact-snapshot WebArena baseline without the not-achievable hint; task prompt difference does not create a new model. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-4 (0613); parameters are undisclosed. GPT-4 (0613)Closed weights · Large · Price proxy Token price: $30 input / $60 output per million. Blended 3:1: $37.5/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. Official default GPT-4 snapshot is 0613; historical model, not GPT-4 Turbo. Checked 2026-10-10 · high confidence | 14.9% | Closed weightsLargePrice proxy | 2023-06 | Source ↗ |
| 34 | gpt-4o-2024-05-13OpenAI · ModelEvaluation & model detailsPublic tracker row from the WebArena team with the not-achievable task hint provided. Tracker explicitly names the May 13 snapshot. Use its legacy token tariff, not later GPT-4o pricing. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-4o (2024-05-13); parameters are undisclosed. GPT-4o (2024-05-13)Closed weights · Large · Price proxy Token price: $5 input / $15 output per million. Blended 3:1: $7.5/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. Legacy standard table explicitly prices this May snapshot at $5/$15. Do not apply the later GPT-4o alias's $2.50/$10 rates. Checked 2026-10-10 · high confidence | 13.05% | Closed weightsLargePrice proxy | 2024-05 | Source ↗ |
| 35 | gpt-4-0613 (with not-achievable hint)OpenAI · ModelEvaluation & model detailsOriginal WebArena GPT-4 baseline when the not-achievable task hint is provided. Original exact-snapshot WebArena baseline with the not-achievable hint; shared checkpoint with the no-hint row. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-4 (0613); parameters are undisclosed. GPT-4 (0613)Closed weights · Large · Price proxy Token price: $30 input / $60 output per million. Blended 3:1: $37.5/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. Official default GPT-4 snapshot is 0613; historical model, not GPT-4 Turbo. Checked 2026-10-10 · high confidence | 11.7% | Closed weightsLargePrice proxy | 2023-06 | Source ↗ |
| 36 | Patel et al. (Qwen1.5-72B, Mixture B)Academic Research · ModelEvaluation & model detailsTable 3 reports 9.36% for the Qwen1.5-72B policy fine-tuned on synthetic Mixture B; GPT-4 generated reference trajectories for analysis. Paper Table 3 assigns 9.36% to Qwen1.5-72B fine-tuned on synthetic Mixture B. GPT-4 generated reference trajectories for evaluation analysis, not the runtime policy. No matching evaluated fine-tune release was verified. Harness code: uncertain. Published size: from the parameter count of Patel et al. Qwen1.5-72B Mixture-B. Patel et al. Qwen1.5-72B Mixture-BUncertain weights · Medium · Published size · 72B total The paper's 9.36 result belongs to Qwen1.5-72B-Chat fine-tuned on synthetic Mixture B. GPT-4 supplies reference trajectories for analysis, not this policy. Exact fine-tuned weights were not located in the source, so availability is unresolved. Checked 2026-10-10 · medium confidence | 9.36% | Uncertain weightsMedium | 2024-05 | Source ↗ |
| 37 | gpt-3.5-turbo-16k-0613OpenAI · ModelEvaluation & model detailsOriginal WebArena GPT-3.5 baseline. Original exact 16K snapshot. Its historical $3/$4 tariff puts it in Large under the specified price proxy, despite the product generation. Harness code: open. Price proxy: the size class comes from the standard token price of GPT-3.5 Turbo 16K (0613); parameters are undisclosed. GPT-3.5 Turbo 16K (0613)Closed weights · Large · Price proxy Token price: $3 input / $4 output per million. Blended 3:1: $3.25/M. Parameters are undisclosed. Standard uncached USD text token prices per million determine the category; tool fees and cache/batch discounts are excluded. Historical deprecation table explicitly lists this exact 16K snapshot at $3/$4. The cheaper modern GPT-3.5 alias tariff is not substituted. Checked 2026-10-10 · high confidence | 8.87% | Closed weightsLargePrice proxy | 2023-03 | Source ↗ |
| 38 | Qwen-1.5-chat-72bQwen · ModelEvaluation & model detailsPatel et al. tracker row for the open-weight Qwen 1.5 chat model. Base Qwen1.5-72B-Chat result from the self-improvement study; not its fine-tuned Mixture-B model. Harness code: uncertain. Published size: from the parameter count of Qwen1.5-72B-Chat. Qwen1.5-72B-ChatOpen weights · Medium · Published size · 72B total Published 72B dense chat model; distinct from Patel et al.'s subsequently fine-tuned Mixture-B checkpoint. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 7.14% | Open weightsMedium | 2023-09 | Source ↗Repository ↗ |
| 39 | Gemini ProGoogle · ModelEvaluation & model detailsWebArena tracker baseline for Gemini Pro; implementation is proprietary. December 2023 Gemini Pro tracker baseline. Model name maps to original 1.0 Pro, not 1.5/2.x/3.x Pro; exact snapshot and token rates are unresolved. Harness code: open. Estimated for Gemini 1.0 Pro (historical WebArena model): no verified parameter count or standard token price, so the size class is provisional. Gemini 1.0 Pro (historical WebArena model)Closed weights · Medium · Estimated Original Gemini Pro identity follows the December 2023 tracker entry. The current official legacy table retains $0.000125 input / $0.000375 output per 1,000 characters, not tokens. This lower-priced legacy tier supports a provisional Medium category; no exact token conversion or parameter count is asserted. Checked 2026-10-10 · low confidence | 7.12% | Closed weightsMediumEstimated | 2023-12 | Source ↗ |
| 40 | Llama3-chat-70bMeta · ModelEvaluation & model detailsPublic tracker baseline for the open-weight Llama 3 70B Instruct model. Tracker names original Llama 3 chat 70B; public Instruct checkpoint is the matching instruction variant. Harness code: open. Published size: from the parameter count of Meta Llama 3 70B Instruct. Meta Llama 3 70B InstructOpen weights · Medium · Published size · 70B total Original Llama 3 instruction checkpoint; not Llama 3.1. Actual checkpoint files were verified in the public repository inventory. Access requires the publisher's license acceptance; this is released weight access, not an unrestricted license. Checked 2026-10-10 · high confidence | 7.02% | Open weightsMedium | 2024-04-02 | Source ↗Repository ↗ |
| 41 | Synatra-CodeLLama7bAcademic Research · ModelEvaluation & model detailsSynatra CodeLlama-7B WebArena row from the paper and public tracker. Exact Synatra web-agent fine-tune; public official repository links its actual checkpoint and WebArena evaluation code. Harness code: open. Published size: from the parameter count of Synatra-CodeLlama-7B. Synatra-CodeLlama-7BOpen weights · Small · Published size · 7B total The official Synatra repository links this publisher checkpoint; its Synatra-CodeLlama subdirectory contains three safetensors shards. Unrelated similarly named Synatra chat models are excluded. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 6.28% | Open weightsSmall | 2024-10 | Source ↗ |
| 42 | Lemur-chat-70bOpenLemur · ModelEvaluation & model detailsLemur-chat-70B WebArena row; open model weights are available on Hugging Face. Lemur paper identifies the 70B chat-v1 checkpoint and publishes weights; no additional learned runtime model is established. Harness code: open. Published size: from the parameter count of Lemur-70B-Chat-v1. Lemur-70B-Chat-v1Open weights · Medium · Published size · 70B total Published 70B code/text model; public PyTorch weight shards verified. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 5.3% | Open weightsMedium | 2023-10 | Source ↗Repository ↗ |
| 43 | Agent-FLANInternLM · ModelEvaluation & model detailsAgent-FLAN result from the paper and public tracker; code and model artifacts are public. The 7B Agent-FLAN checkpoint is released and matches the paper's Llama2-7B primary experiment. Teacher models for training are excluded. Harness code: open. Published size: from the parameter count of Agent-FLAN-7B. Agent-FLAN-7BOpen weights · Small · Published size · 7B total Llama2-7B fine-tuned on Agent-FLAN; the exact released model card identifies this base. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 4.68% | Open weightsSmall | 2024-03 | Source ↗Repository ↗ |
| 44 | CodeLlama-instruct-34bMeta · ModelEvaluation & model detailsLemur paper baseline for the public Code Llama 34B Instruct model. Original Code Llama 34B instruction model in the Lemur evaluation; nominal total size determines Small. Harness code: open. Published size: from the parameter count of Code Llama 34B Instruct. Code Llama 34B InstructOpen weights · Small · Published size · 34B total Published nominal dense model size. Original publisher inventory and compatible codellama mirror contain actual weight shards. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 4.06% | Open weightsSmall | 2023-08 | Source ↗Repository ↗ |
| 45 | AgentLM-70bTHUDM · ModelEvaluation & model detailsAgentTuning paper reports the AgentLM-70B WebArena result and links released AgentLM models. Released AgentLM-70B. GPT-4 used to produce training trajectories is excluded from inference. Harness code: open. Published size: from the parameter count of AgentLM-70B. AgentLM-70BOpen weights · Medium · Published size · 70B total Llama2-chat fine-tuned with AgentInstruct plus general conversation data. Training teachers are excluded. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 3.81% | Open weightsMedium | 2023-10 | Source ↗Repository ↗ |
| 46 | Llama3-chat-8bMeta · ModelEvaluation & model detailsPublic tracker baseline for the open-weight Llama 3 8B Instruct model. Original Llama 3 8B Instruct checkpoint; not the Llama-3.1 NNetNav fine-tune. Harness code: open. Published size: from the parameter count of Meta Llama 3 8B Instruct. Meta Llama 3 8B InstructOpen weights · Small · Published size · 8B total Original Llama 3 instruction checkpoint; not Llama 3.1. Actual checkpoint files were verified in the public repository inventory. Access requires the publisher's license acceptance; this is released weight access, not an unrestricted license. Checked 2026-10-10 · high confidence | 3.32% | Open weightsSmall | 2024-04 | Source ↗Repository ↗ |
| 47 | CodeAct AgentAcademic Research · ModelEvaluation & model detailsSynatra reevaluation reports 2.34%, rounded here to 2.3%; the exact evaluated 7B CodeAct checkpoint remains unspecified. The matching 2.34 result is a Synatra reevaluation of CodeActAgent-7B with a generic action space. The original CodeAct paper offers both Llama2 and Mistral 7B variants; this row's exact checkpoint was not verified. The public Synatra implementation does not expose a CodeAct evaluation configuration, so exact reevaluation harness availability is unverified. Harness code: uncertain. Published size: from the parameter count of CodeActAgent 7B (WebArena variant unpinned). CodeActAgent 7B (WebArena variant unpinned)Uncertain weights · Small · Published size · 7B total Synatra Table 1 reports 2.34 for CodeActAgent-7B but does not resolve Llama2 versus Mistral. Both original 7B variants have released weights; exact evaluated variant remains unpinned, so this record does not assert a particular checkpoint.
Checked 2026-10-10 · medium confidence | 2.3% | Uncertain weightsSmall | 2024-02 | Source ↗Repository ↗ |
| 48 | AgentLM-13bTHUDM · ModelEvaluation & model detailsAgentTuning paper reports the AgentLM-13B WebArena result and links released AgentLM models. Released AgentLM-13B; model size is distinct from the 7B and 70B rows. Harness code: open. Published size: from the parameter count of AgentLM-13B. AgentLM-13BOpen weights · Small · Published size · 13B total Llama2-chat fine-tuned with AgentInstruct plus general conversation data. Training teachers are excluded. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 1.6% | Open weightsSmall | 2023-10 | Source ↗Repository ↗ |
| 49 | Mixtral-8x7BMistral AI · ModelEvaluation & model detailsPublic tracker baseline for the open-weight Mixtral-8x7B Instruct model. Tracker links Mixtral-8x7B-Instruct-v0.1. Its 46.7B total places it in Medium; 12.9B active is not used for size classification. Harness code: open. Published size: from the parameter count of Mixtral-8x7B-Instruct-v0.1. Mixtral-8x7B-Instruct-v0.1Open weights · Medium · Published size · 46.7B total · 12.9B active The publisher states 46.7B total and 12.9B active. Category uses the total, not 7B, active parameters, or eight times the name. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 1.39% | Open weightsMedium | 2024-01 | Source ↗Repository ↗ |
| 50 | AgentLM-7bTHUDM · ModelEvaluation & model detailsAgentTuning paper reports the AgentLM-7B WebArena result and links released AgentLM models. Released AgentLM-7B; GPT training teachers are excluded. Harness code: open. Published size: from the parameter count of AgentLM-7B. AgentLM-7BOpen weights · Small · Published size · 7B total Llama2-chat fine-tuned with AgentInstruct plus general conversation data. Training teachers are excluded. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 0.74% | Open weightsSmall | 2023-10 | Source ↗Repository ↗ |
| 51 | FireActAcademic Research · ModelEvaluation & model detailsAgent-FLAN reevaluation reports 0.25%; the exact evaluated 7B FireAct checkpoint remains unspecified. The original FireAct paper evaluates QA rather than this WebArena result; Agent-FLAN Table 1 reports the matching 0.25 for FireAct-7B. The exact full/adapter checkpoint for that reevaluation is unpinned. Harness code: uncertain. Published size: from the parameter count of FireAct 7B (WebArena variant unpinned). FireAct 7B (WebArena variant unpinned)Uncertain weights · Small · Published size · 7B total The 0.25 WebArena result is reported by Agent-FLAN, not the original QA-focused FireAct paper. Inventories verify the released full Llama2-7B checkpoint and Llama2/CodeLlama-7B LoRA variants, but the exact artifact used for this third-party WebArena run is unpinned. GPT-4 was a training teacher, not a required inference component. Checked 2026-10-10 · medium confidence | 0.25% | Uncertain weightsSmall | 2023-10 | Source ↗ |
| 52 | CodeLlama-instruct-7bMeta · ModelEvaluation & model detailsPublic tracker baseline for the public Code Llama 7B Instruct model. Tracker explicitly names the released Code Llama 7B Instruct model; zero score does not imply missing model metadata. Harness code: open. Published size: from the parameter count of Code Llama 7B Instruct. Code Llama 7B InstructOpen weights · Small · Published size · 7B total Published nominal dense model size. Original publisher inventory and compatible codellama mirror contain actual weight shards. Actual checkpoint files were verified in the public repository inventory. Checked 2026-10-10 · high confidence | 0.0% | Open weightsSmall | 2023-08 | Source ↗Repository ↗ |
About this benchmark
WebArena evaluates browser agents in reproducible, self-hosted websites instead of the open live web. Its 812 tasks cover e-commerce, forum discussion, collaborative software development, content management, maps, and reference lookup.
The benchmark is strongest when you care about repeatable web-agent experiments: every task has a controlled start state and functional success criteria rather than a changing production website.
Because many rows come from a public community tracker, a WebArena score should be read alongside the source, submitted scaffold, observation mode, and whether the result was independently reproduced.
Controlled environments improve reproducibility, but tracker rows still vary by scaffold and submission policy.
Filtered task-set or modified-grader reports are not ranked as full WebArena results unless the row notes that setup explicitly.
Example tasks
Three public tasks quoted from benchmark sources:
- "What is the top-1 best-selling product in 2022" Citation: WebArena test config
- "Tell me the full address of all international airports that are within a driving distance of 50 km to Carnegie Mellon University" Citation: WebArena test config
- "Tell me the the number of reviews that our store received by far that mention term "disappointed"" Citation: WebArena test config
Methodology
- Primary metric is end-to-end task success rate on the WebArena task set; the original GPT-4-based baseline was 14.41% versus 78.24% human performance.
- Evaluation checks functional correctness through task-specific validators and answer checks in the hosted environment.
- Scores can change with prompt scaffolding, observation mode, browser action space, and retry or step budget.
- We prefer rows tied to WebArena's public leaderboard, papers, or repositories that include enough setup detail to reproduce the run.
Links
Related benchmarks
Compare this benchmark with related pages from the hub: