VulcanBench

--network (off by default; opt in for dependency installs)

```

- docker (default) runs tools in a non-root, network-off, resource-limited container (workspace bind-mounted, cleaned up after each run). It errors out if the daemon is unreachable — it never silently falls back to host execution. - local runs the model's commands directly on the host — fast and Docker-free, but unsandboxed; opt in deliberately (fine for mock:synthetic and trusted dev loops). - auto uses Docker when available. Falling back to host execution additionally requires VULCANBENCH_ALLOW_HOST_EXEC=1; otherwise it errors out.

File operations (read/edit/search) always run host-side over the shared mount; command execution (run_command/run_tests/run_lint) and the functional verifier run inside the container, so the whole run is reproduced in one isolated environment. Build the all-language image with `docker build -t vulcanbench/sandbox:base -f sandbox/Dockerfile.base .` (Python + Go + Node).

One-command setup

git clone https://github.com/morganlinton/VulcanBench.git
cd VulcanBench
make setup
source .venv/bin/activate
vulcanbench --help

Dashboard + backend (the dashboard reads live data from the API):

pip install -e ".[backend]"
uvicorn backend.app:app --port 8000          # serves ./runs at /api/*
cd dashboard && npm install && npm run dev    # http://localhost:3000

By default the API reads ./runs/ directly. For a durable, queryable store, set DATABASE_URL (Postgres or SQLite) and the API switches to a database — POST /api/runs and /api/feedback become writable, and python scripts/ingest_runs.py bulk-loads existing runs. A Postgres is provided by docker compose up db.

Real runs default to the Docker sandbox; --sandbox local is fine for the

build the image once with `make sandbox-image`.

Build the base image once (git, ripgrep, ruff, bandit, radon, pytest):

docker build -t vulcanbench/sandbox:base -f sandbox/Dockerfile.base .

vulcanbench run --task hello-world --model openai:gpt-4o

--sandbox local|docker|auto (default: docker)

Example run

```bash

Provider terms & data usage

VulcanBench is an independent evaluation harness. A few boundaries keep its use consistent with the model providers' terms — please read these before running or publishing results.

- You bring your own keys, under your own agreement. VulcanBench never bundles or shares API credentials. Each run uses the keys in your environment (OPENAI_API_KEY, ANTHROPIC_API_KEY, ZAI_API_KEY), so every call is made under your account and your commercial/API agreement with that provider. You are responsible for staying within your provider's terms and usage policies.

- Outputs are for evaluation, not training. Recorded run artifacts (traces, patches, summaries) capture model outputs solely for scoring, inspection, and reproducibility. Both OpenAI and Anthropic prohibit using their outputs to develop or train competing models — do not use VulcanBench artifacts, or any published corpus of them, for that purpose. VulcanBench intentionally has no "export outputs as a training dataset" feature.

- Trademarks & independence. "OpenAI" and "GPT" are trademarks of OpenAI; "Anthropic" and "Claude" are trademarks of Anthropic; "Z.ai" and "GLM" are trademarks of Zhipu AI. VulcanBench is not affiliated with, sponsored by, or endorsed by any of these companies. Model and provider names are used only to identify the systems under test.

This is not legal advice; consult the current provider terms for authoritative guidance.

Offline, deterministic (no API key) — drives the real agent loop end to end.

Any real model via the generic provider interface:

export OPENAI_API_KEY=... # or ANTHROPIC_API_KEY=... vulcanbench run --task hello-world --model openai:gpt-4o vulcanbench run --task hello-world --model anthropic:claude-opus-4-8 vulcanbench run --task hello-world --model zai:glm-5.2

Compare normalized reasoning effort on the same suite/model.

vulcanbench run --suite v1 --model openai:gpt-4o --effort low vulcanbench effort-sweep --suite v1 --model openai:gpt-5.1 --efforts low,medium,high --repeat 3 --sandbox docker

Fast micro/small sweep vs navigation-heavy medium/large tasks:

vulcanbench run --suite v1-micro --model openai:gpt-4o vulcanbench run --suite v1-large --model openai:gpt-4o --repeat 5 --sandbox docker vulcanbench leaderboard # by model: pass@1 ± stderr, pass@k, cost, latency vulcanbench leaderboard --by run # per-run drill-down vulcanbench report -o report.md # shareable Markdown/JSON report (ranking, # model-separation/discrimination, effort # sensitivity, per-task breakdown, # environment, drift flags) vulcanbench calibrate # empirical difficulty calibration from recorded runs vulcanbench replay <id>