AgenticSWE-bench Verified
Human-validated 500-task subset of SWE-bench: real Python repository bug fixes graded by hidden test suites.
From $0.03 / taskRun →
Every benchmark, ready to run.
274 benchmarks on any model, including your own. 107 run end to end in a sandbox.
- Agentic
AgenticSWE-Bench Pro
Real-world software-engineering tasks from open-source repos.
From $0.27 / taskRun →
AgenticTerminal-Bench 2.1
Verified iteration of Terminal-Bench 2.0 (89 tasks): 26 tasks fixed for bugs, timeouts/resources, and reward-hacking robustness (with Z.ai).
From $0.04 / taskRun →
AgenticOSWorld
OSWorld (Xie et al., NeurIPS 2024): open-ended computer-use tasks on the official reference Ubuntu desktop VM, with screenshot+a11y-tree observations and the official per-task grader.
From $0.11 / taskRun →
Agentictau-bench
Tool-use customer-service agent benchmark from Sierra Research, run through the official tau2-bench harness.
From $1.99 / taskRun →
Single turnHumanity's Last Exam (text)
Humanity's Last Exam (Center for AI Safety / Scale, 2025), text-only subset: closed-ended, expert-level questions across dozens of fields.
From $0.01 / taskRun →
AgenticTheAgentCompany
Agentic knowledge-work benchmark: the agent operates inside a simulated software company and completes finance, HR, admin, marketing, project-management, data, and engineering tasks against live self…
From $0.20 / taskRun →
AgenticSWE-Lancer Diamond
Freelance software-engineering benchmark (OpenAI, arXiv:2502.12115): the Diamond set of real Upwork tasks from the Expensify monorepo, each worth a real-world payout.
From $0.06 / taskRun →
AgenticDeepSWE
Long-horizon agentic coding benchmark (Datacurve, 2026): 113 original, contamination-free enhancement/bug-fix tasks across Go, Python, TypeScript, JavaScript, and Rust on active open-source repos, gr…
From $0.20 / taskRun →- Single turn
$OneMillion-Bench
$OneMillion-Bench (humanlaya/OneMillion-Bench, arXiv:2603.07980): 400 expert-authored professional questions across 5 domains (economics & finance, healthcare & medicine, industry, law, natural scien…
From $0.15 / taskRun → 
Single turnAA-LCR
AA-LCR (Artificial Analysis Long Context Reasoning, v1.1): 100 hard questions that need reasoning across several real documents (company and industry reports, government consultations, academic, lega…
From $0.13 / taskRun →- Single turn
AA-Omniscience
AA-Omniscience (Artificial Analysis, arXiv:2511.13029): knowledge + hallucination across 6 economically significant domains and ~40 topics.
From <$0.01 / taskRun → - Single turn
AA-Omniscience v1.1
AA-Omniscience Public v1.1 (Artificial Analysis, 2026-10-27): knowledge + hallucination across 6 economically significant domains.
Science & ML researchRun → 
AgenticAgentDojo (important_instructions attack: utility under attack)
AgentDojo (ETH Zurich SPY Lab, NeurIPS 2024; ethz-spylab/agentdojo, MIT), benchmark version v1.2.2: realistic tool-use tasks in four simulated environments (workspace email/calendar/drive, travel, ba…
From <$0.01 / taskRun →- Agentic
AgentDojo (no attack: utility)
AgentDojo (ETH Zurich SPY Lab, NeurIPS 2024; ethz-spylab/agentdojo, MIT), benchmark version v1.2.2: realistic tool-use tasks in four simulated environments (workspace email/calendar/drive, travel, ba…
From <$0.01 / taskRun → - Agentic
Agents' Last Exam (ALE-CLI)
Agents' Last Exam v1.1 (UC Berkeley RDI), the ALE-CLI track: the 86 Linux tasks (28 near-term, 38 full-spectrum, 23 last-exam; tiers overlap) of long-horizon professional work on a real Ubuntu 22.04…
From $0.76 / taskRun → - Single turn
AGIEval
AGIEval-en (Zhong et al. 2023), v1.1 data: all ten English tasks — AQuA-RAT, LogiQA-en, LSAT-AR/LR/RC, SAT-En, SAT-En without passage, SAT-Math, Gaokao-English and MATH (cloze).
From <$0.01 / taskRun → - Single turn
AI2D
Grade-school science diagram MCQ (Kembhavi et al. 2016): answer a question about a diagram by choosing among text options.
From <$0.01 / taskRun → - Agentic
Aider Polyglot
225 Exercism-style code-editing exercises across Python, JavaScript, Java, Go, Rust, and C++ (Aider-AI/polyglot-benchmark) with hidden tests.
From <$0.01 / taskRun → 
AgenticAndroidWorld
AndroidWorld (Google Research, ICLR 2025): the 116 official task templates across 20 Android apps on a live Pixel 6 / API 33 emulator, one instance each at the official seed (30), driven by the paper…
From $0.08 / taskRun →- Single turn
ARC Challenge
AI2 Reasoning Challenge (hard split).
From <$0.01 / taskRun → - Single turn
ARC Easy
AI2 Reasoning Challenge (easy split).
From <$0.01 / taskRun → - Single turn
ARC-AGI-1 (public eval)
ARC-AGI-1 abstract reasoning (Chollet 2019): the 400-task public evaluation set (419 test pairs), read from the canonical fchollet/ARC-AGI repository at a pinned commit and verified by content digest.
From <$0.01 / taskRun → - Single turn
ARC-AGI-2
ARC-AGI-2 abstract reasoning (Chollet et al. 2025), via the arc-agi-community mirror.
From $0.02 / taskRun →