Your agent reads the screen as stable element ids, not pixels, verifies every action, and routes it through one safety gate. Local, any model.
$ Then run clawdcursor consent --accept. On macOS also run clawdcursor grant. Done.
Same tools, two entry shapes. Pick once at install.
AI lives in your editor. It spawns clawdcursor over stdio. No daemon, no port.
{
"mcpServers": {
"clawdcursor": {
"command": "clawdcursor",
"args": ["mcp", "--compact"]
}
}
}HTTP MCP on 127.0.0.1:3847/mcp. doctor then agent runs the built-in autonomous loop; agent --no-llm serves tools only when your agent has its own brain.
A11y tree before pixels. Vision only when needed.
Fuse the a11y tree + OCR into one confidence-scored el_NN map. Act on an element by stable id. No image bytes to the model. Vision is last resort.
OCR when the tree is sparse. A screenshot only when you truly need pixels: canvas-only apps or spatial reasoning.
Pass expect and the action confirms its outcome, reporting a DEVIATION if the UI didn't obey. Every call routes through one safety layer.
Screenshot-based computer use sends the model a picture of the screen every time it looks. clawdcursor sends text the model can act on by name.
| Screenshot-based computer use | clawdcursor | |
|---|---|---|
| What the model reads per look | A screenshot: ~1,000–1,800 tokens | Text: 38 tokens to find a button, 50 to read a field |
| How it clicks | x, y coordinates guessed from the picture | By name: invoke "Equals". Coordinates are the fallback |
| Did the click work? | Take another screenshot | expect checks it in the same call and reports a DEVIATION |
| Known steps in a row | A model turn + a screenshot per step | One batch call, every step still safety-checked |
| Reading text | Read off the pixels | The app's own text: tree values, copy_all_text, OCR |
| Which model | One that can read screenshots and point at pixels | Any model with tool calling, 13+ providers |
Measured on Windows 11 (October 2026): 127 × 49 in Calculator took clawdcursor 3 tool calls and ~176 tokens of screen input, with 0 screenshots. A screenshot loop needs 2 to 8 screenshots for the same task (2,072–8,288 tokens at 1024×768). Screenshot cost from Anthropic's computer use docs; clawdcursor tokens estimated as characters ÷ 3.5.
Windows x64/ARM64 · macOS 12+ · Linux X11/Wayland
Platform-aware key combos. Cmd on macOS, Ctrl elsewhere. No LLM cost.
Collapse N deterministic tool calls into a single guarded, safety-gated batch. N calls → 1.
At a sign-in, 2FA code or CAPTCHA the agent pauses instead of guessing. The glow turns blue; click Resume and it carries on, or "Something else" to type it an answer.
A banner and an amber screen-edge glow on Windows, macOS and Linux while an agent drives. Double-click to stop.
A native accessibility helper: 20 windows listed in 0.3 s, a window's tree read in 0.06–0.2 s.
| Compound | Purpose | Actions |
|---|---|---|
computer |
Mouse, keyboard, screenshots. Raw I/O. | screenshot · click · double_click · right_click · triple_click · hover · scroll · scroll_horizontal · drag · drag_path · type · key · wait |
accessibility |
Drive UI by element name, not pixel. Survives DPI, resize, layout shifts. | read_tree · find · get_element · focused · invoke · focus · set_value · get_value · expand · collapse · toggle · select · state · list_children · wait_for |
window |
Launch, focus, resize. App-level state. | list · active · focus · maximize · minimize · restore · close · resize · list_displays · screen_size · open_app · open_file · open_url · switch_tab · navigate |
system |
Clipboard, OCR, shortcuts, undo, webview detection, CDP relaunch, task delegation. The meta surface for an external brain. | clipboard_read · clipboard_write · system_time · ocr · undo · shortcuts_list · shortcuts_run · delegate · detect_webview · relaunch_with_cdp · system_prompt · pause_for_user |
browser |
Chrome DevTools Protocol: real DOM access for Electron / WebView2 apps whose a11y tree is sparse. | connect · page_context · read_text · click · type · select_option · evaluate · wait_for · list_tabs · switch_tab · scroll |
task |
Hand the whole task to the autonomous loop. Daemon mode only: needs clawdcursor agent with an LLM configured. |
run · status · abort · resume |
Compact (~7,000 tokens of catalog, a third of granular): computer({ "action": "key", "combo": "mod+s" }).
Granular: key_press({ "key": "mod+s" }).
Both hit the same safety.evaluate() chokepoint. Pass --granular for the granular surface.
See schema.snapshot.json for every parameter.
# Install & setup
clawdcursor consent # one-time desktop-control authorization (always required)
clawdcursor grant # macOS only: Accessibility + Screen Recording prompts. MCP setup ends here.
clawdcursor doctor # ONLY for `agent` mode: configures the daemon's built-in LLM (+ diagnostics)
clawdcursor status # readiness check (consent, permissions, AI config)
clawdcursor versions # every installed copy and its version
# Run
clawdcursor mcp # stdio MCP server for editor hosts
clawdcursor mcp --compact # same, with 7 compound tools (recommended)
clawdcursor agent # HTTP MCP daemon at :3847/mcp, optional built-in LLM
clawdcursor agent --no-llm # tool surface only: your agent brings its own brain
clawdcursor resume # continue a task paused for you (--end to stop it)
clawdcursor stop # stop every running mode
clawdcursor uninstall # remove all clawdcursor config and dataOpen source. Any model. Localhost-only by default, enforced. No telemetry.
Star on GitHub