ClawBench

<img src="static/icons/circle-question.svg" width="28" height="28"> Example Walkthrough

Curious what one task actually looks like, start to finish? Here's task 001 end to end.

The task — from test-cases/v1/001-daily-life-food-uber-eats/task.json:

{
  "instruction": "On Uber Eats, order delivery: one Pad Thai, deliver to home address, note \"no peanuts\"",
  "time_limit": 30,
  "eval_schema": {
    "url_pattern": "__PLACEHOLDER_WILL_NOT_MATCH__",
    "method": "POST"
  }
}

The agent gets this instruction verbatim, plus read-only access to /my-info/alex_green_personal_info.json (the dummy user's name, home address, phone, date of birth) and a disposable email account for any sign-in prompt. It has 30 minutes to reach a POST request — any longer and the container is killed.

What the agent does (the happy path):

1. Navigates to ubereats.com
2. Reads the dummy user's home address from /my-info/alex_green_personal_info.json and enters it in the delivery-address box
3. Searches for "Pad Thai" in the food search
4. Picks a restaurant that has Pad Thai available for delivery to that address
5. Opens the item detail page, finds the customization or special-instructions field, enters "no peanuts"
6. Adds one to cart, opens the cart, and handles any sign-in prompt using the disposable email credentials
7. Reaches checkout, taps Place Order

What the interceptor catches — that final Place Order tap fires a POST request. ClawBench's request interceptor sits in front of the browser and captures the outbound request before it reaches Uber Eats's servers, so the dummy user is never actually charged. At the exact moment of interception, all five recording layers (MP4 video, PNG screenshots, HTTP traffic, browser actions, agent messages) are frozen into /data/.

How the judge decides PASS / FAIL — task 001's url_pattern is the intentional sentinel __PLACEHOLDER_WILL_NOT_MATCH__, which means no request path can mechanically match. The verdict comes from the agentic judge in eval/agentic_eval.md, which replays the five-layer recording against a human reference run and checks four things:

• Did the agent actually reach the final checkout step?
• Is the cart exactly one Pad Thai (not two, not a combo)?
• Is the delivery address the user's home address from alex_green_personal_info.json?
• Does the order carry the "no peanuts" note in the instructions field?

All four must hold for a PASS. Miss any one and it's a FAIL with evidence from the recording pinned to the failing criterion. This per-task rubric is what makes ClawBench judge-sensitive rather than URL-regex-sensitive — see eval/README.md for the full rubric format and eval/agentic_eval.md for the judge prompt.

<br/>

<img src="static/icons/circle-question.svg" width="28" height="28"> FAQ

<details> <summary><b>What data does each run produce?</b></summary>

Each session records five layers of synchronized data under /data/:

For the Pi harness, agent-messages.jsonl is filtered Pi JSON mode output, including message_start/message_end events, tool_execution_* events, tool-call content blocks, and thinking blocks when the selected model emits reasoning. Streaming message_update fragments, including *_delta rows, are omitted because complete assistant messages are already preserved in message_end events.

Harness diagnostic logs such as Pi's agent.log and proxy.log are not copied into the final data/ directory.

The interceptor result is saved to interception.json.

</details>

<details> <summary><b>How does the request interceptor work?</b></summary>

The interceptor blocks critical, irreversible HTTP requests (checkout, form submit, email send) to prevent real-world side effects. It connects to Chrome via CDP's Fetch domain and matches requests against the eval schema (url_pattern regex + method + optional body/params). When triggered, it saves the blocked request to interception.json, kills the agent, and stops recording.

The interceptor does not validate task completion -- evaluation is handled separately by evaluators post-session.

For tasks behind payment walls (agent has no valid credit card), the eval schema uses a placeholder pattern that never matches, so the session runs until timeout.

</details>

<details> <summary><b>What is the synthetic user profile?</b></summary>

Each container gets a /my-info/ directory with a dummy user identity (Alex Green): personal info JSON, email credentials, and a resume PDF. The email is a fresh disposable PurelyMail address generated per run. The agent reads these files when it needs to fill forms, register accounts, etc.

Source templates: src/clawbench/runtime/shared/alex_green_personal_info.json (profile) and src/clawbench/runner/run_support/resume_template.json (resume).

</details>

<details> <summary><b>Can I use Podman instead of Docker?</b></summary>

Yes. Set export CONTAINER_ENGINE=podman. The framework auto-detects whichever is available. Podman works without root privileges.

</details>

<details> <summary><b>What tools can the agent use?</b></summary>

All supported harnesses run inside the same container recording and interception environment. CLI/MCP harnesses expose the browser tool plus a restricted set of read-only shell commands (ls, cat, find, grep, head, tail, jq, wc, etc.); commands that could bypass the browser (curl, python, node, wget) are blocked. Hermes and Pi use native browser/file tools attached to the same ClawBench Chrome CDP endpoint. The Pi harness intentionally allowlists only read-only file tools and browser interaction tools; bash, write, edit, browser_http_get, and browser_run_script are not enabled. The agent instruction also explicitly requires browser-only task completion.

</details>

<details> <summary><b>How do I add a new test case?</b></summary>

See CONTRIBUTING.md. In short: create a directory under the target corpus (test-cases/v1/ for V1 or test-cases/v2/ for V2) with a task.json conforming to test-cases/task.schema.json, define the eval schema, test with human mode, and submit a PR.

</details>

<br/>

Frequently Asked Questions

What is ClawBench? ClawBench is an open-source benchmark for AI browser agents — the systems (GPT-based, Claude-based, or open) that drive a real web browser to complete a user's task. V1 measures whether the agent actually finishes 153 everyday online tasks across 144 live websites; V2 adds a 130-task corpus in test-cases/v2/. It measures completion, not whether the agent produces the right-looking text.

What kinds of tasks does ClawBench cover? Fifteen life categories: food delivery, travel booking, job applications, shopping, housing search, email and calendar management, academic research, software development, learning platforms, and more. Every task is something a normal person might do in a normal week, on a real website.

Are 153 tasks enough for evaluation? Yes for a V1 benchmark signal: the 153 tasks span 144 live websites and 15 life categories, and each full run is expensive because it uses isolated containers, real websites, five-layer recording, and post-session judgment against human references. V2 adds another 130 tasks in test-cases/v2/. For cheaper iteration, start with the 20-task test-cases/v1-lite/ subset.

How is a task judged successful? Each task runs in an isolated browser container with a five-layer recording: video, screenshots, network requests, browser actions, and agent messages. For the original V1 results, an evaluator compares the agent trajectory against human reference runs and assigns PASS/FAIL with evidence from the recording. For V2 and newer leaderboard rows, scoring is two-stage: first, the request interceptor checks whether the final blocked HTTP request matches the task's URL/method schema; second, an LLM judge checks whether the captured request payload fulfills the natural-language instruction.

How do account login, registration, and initial task state work? Each run receives a synthetic user profile plus a fresh disposable PurelyMail address. If a task requires sign-up, the agent normally starts from scratch and registers during the run, using the provided identity and email. If a task needs starting files or workspace context, those files live under the task's extra_info/ directory and are mounted for the agent at runtime.

What happens when live websites change? Live-site change is part of the benchmark's target: ClawBench measures whether agents can handle production websites rather than frozen snapshots. That also means some runs can be affected by layout changes, availability, anti-bot systems, or alternate flows. Reproducibility comes from publishing task definitions, eval schemas, run metadata, and five-layer traces; repeated runs over time are still useful for measuring site drift.

Do CAPTCHA or bot checks dominate failures? If an agent encounters a CAPTCHA, it must attempt it. We have seen cases where frontier models are able to solve some CAPTCHAS. CAPTCHA failures can reflect model behavior, browser-control stack limits, or site defenses. The trace datasets make these failures inspectable.

What's the current top score? 33.3% — roughly one task in three — from the strongest frontier model we evaluated. The majority of tasks still defeat every model we've tested; the headroom is real, and the benchmark is not saturated.

Which harness are the published model results based on? The repo default is openclaw, but leaderboard rows include their harness explicitly. V1 results used OpenClaw; newer runs may use Hermes or other supported harnesses. Use the harness column when comparing models, because model and harness changes are separate experimental axes.

Is ClawBench tightly coupled to OpenClaw? No. OpenClaw is the default harness, but ClawBench supports interchangeable harnesses listed in src/clawbench/runtime/harnesses/.

Can ClawBench evaluate CLI agents? Yes. ClawBench is a browser-task benchmark, but CLI and coding-agent harnesses can drive the same instrumented Chromium session using native tools or MCPs.

How do I reproduce a published score? From a source checkout, configure models/models.yaml, then run uv run clawbench. The TUI builds the container image and runs local tasks against your model of choice. For batch runs, use --all-cases for the default V1 suite, --cases-suite v2 --all-cases for V2, or --cases-suite v1-lite --all-cases for Lite.

Will newer models be added? Yes. New model runs can be submitted or requested through the contribution flow and issues. Public rows are added as complete or clearly marked partial runs, depending on what has finished.

Is ClawBench safe to run against live websites? The runner uses a hardened container with a request interceptor that blocks purchases, account creation, outbound email sends, and similar irreversible actions by default. Tasks that need to simulate those actions (e.g., "add to cart and checkout") terminate at the last reversible step. You can relax the interceptor per-task if your research requires it.

Can I contribute new tasks or harnesses? Yes. V1 tasks live in test-cases/v1/; V2 tasks live in test-cases/v2/; Lite tasks live in test-cases/v1-lite/. See CONTRIBUTING.md for the task schema and validation flow.

How does ClawBench relate to HarnessBench? Same scoring pipeline, orthogonal axis. ClawBench fixes the harness and varies the model; HarnessBench fixes the model and varies the harness. They share the V1 153-task corpus, the five-layer recording, and the agentic evaluator — so numbers are directly comparable.