computer-use-mcp

01What it is

Most "computer use" runs against a sandboxed VM or a remote framebuffer: the agent gets one composited pixel stream and a virtual mouse. This project is different — it's a local, trusted, in-session driver. Because it runs on your real Mac as you, it can reach below the pixels:

02The big idea: two operators, no focus fight

The central promise — the thing every subsystem exists to protect — is background control. You keep typing in your foreground app while an agent drives a different app that may be occluded behind other windows. The agent's synthetic clicks and keystrokes never move your real cursor and never steal keyboard focus.

Three principles fall out of that promise, and they're the lens for everything below:

03Process model

One binary, launched as different subcommands (see main.swift). Understanding the three-process split is the single most important thing about this codebase.

Why a shared daemon?

• No collisions by construction. Capture, AX, and input are single-owner. Multiple agents can't stomp each other's screenshots or event streams.
• Cross-process element ids. Snapshots persist to disk, so an element_id minted in one process resolves in another (serve, the call harness, or a restarted server).
• App leases. The daemon serializes action tools against the same resolved pid for a short window — two sessions can't interleave actions in one app. Perception is never blocked. Daemon/AppLeases.swift

Other subcommands are operational: doctor (permission check/prompt), health_report (identity + permissions + capture + daemon diagnostics), call (invoke one tool from the shell — the dev/self-test harness), version.

04A request's journey

Every entry point (serve, daemon, call) converges on one dispatch funnel, dispatchTool() in Dispatch.swift. That single funnel is why guarantees are uniform — nothing reaches a handler without passing the same gates.

// client → serve → daemon → dispatchTool()
1. catalog lookup            // unknown tool → error
2. RateLimiter.acquire()     // optional global throttle (off by default)
3. SleepAssertion.noteActivity()  // keep the Mac awake while calls flow
4. preflightRefusal():
      ├─ screen-lock pause     // mutating tools pause when locked
      ├─ interference yield    // yield if the human is active in this app
      └─ URL policy            // browser deny/confirm patterns
5. spec.handler(arguments)   // the actual tool logic
6. log + Telemetry.record    // per-call stderr line + counters

At the daemon level there are two more gates in front of this: peer auth (same-UID + bearer token, DaemonServer.swift:239) and app-lease check before dispatch. The result comes back carrying _meta blocks — focus telemetry and the outcome verdict (§7) — that generic MCP clients can read to detect regressions.

05Perception & snapshots — live AX identity

How the agent "sees". get_app_state is the canonical surface: it returns one window's AX tree + a ScreenCaptureKit screenshot, with every element boxed in the screenshot's pixel space. Tools/Perception.swift · Core/Screenshot.swift

The stale-reference problem, solved with exact handles

The daemon gives the model an element_id backed by the exact live AXUIElement captured for that node. Before an action, it proves the handle is live, compares its semantic fingerprint, checks the owning PID, and proves attachment to the captured window. Core/Snapshot.swift · Core/Target.swift

Three coordinate spaces (name which one, always)

Coordinate actions require a prior snapshot (it carries window origin, scale, pixel bounds). Out-of-bounds coordinates are rejected, not clipped. Core/HitTest.swift

Cheap loops: diffs, not re-sends

After a mutating action, the result returns reduced fresh state. If the tree didn't change, it says "existing ids remain valid" instead of resending. If it did, it returns a compact diff — changed (~), appeared (+), disappeared (−) — falling back to the full tree only when >50% churned. For sparse UIs, use ocr:true; for truncated trees, scope get_app_state to a retained container. Core/TreeBuilder.swift · Core/OCR.swift

06The dispatch ladder — how it acts without stealing focus

This is the heart of the background guarantee. For any click/scroll/key, the server tries delivery paths AX-first, event-last, and — critically — never auto-escalates past tier 3, because macOS gives no reliable "event landed" signal. Core/Input.swift

07Verifier-first outcomes

The design pivot that makes background control trustworthy: move mutating tools from "success = the AX call didn't throw" to "success = the effect was observed." Core/ActionOutcome.swift · docs/outcome-contract.md

Every mutating tool snapshots the target element's fields before dispatch, re-reads them (plus the whole-window change bit) after, diffs the fields that matter for that action family, and reduces to a four-value verdict carried in a computer-use-mcp/outcome _meta block:

Note isError is left unchanged — a verified no-effect is not a throw, it's data. This composes with the focus telemetry block (focus_changed, delivery_tier, ui_changed, frontmost before/after) from Core/FocusTelemetry.swift. A generic client treating focus_changed:true + focus_change_allowed:false as a regression is the intended contract.

08Safety gates the agent can't bypass

These run server-side inside the funnel — no tool argument disables them casually. Core/SafetyPolicy.swift · InterferenceGuard.swift · URLPolicy.swift · SessionPower.swift

09The agent cursor overlay

Purely cosmetic, but it's what makes background control legible to a watching human: you can see where the agent is working without it touching your real pointer. Overlay/OverlayHelper.swift · AgentCursor.swift

• Separate process because the MCP server is a headless async stdio process with no AppKit run loop. The helper runs an .accessory NSApplication — no Dock icon, never frontmost.
• Singleton — one cursor serves every concurrent server process. Reads target points from a shared FIFO (move <globalX> <globalY>, ping keep-alives) and glides a glyph toward them.
• Multi-display correct. One click-through borderless panel per display (because "Displays have separate Spaces" clips a window to one screen). The cursor position is modeled once in AppKit global coords and mirrored into every panel; each clips to its own screen, so the glyph appears on whichever display holds it — even straddling a boundary mid-glide.

10Tool catalog — 17 tools

Schemas + model-facing descriptions live in Tools/Catalog.swift; handlers in the sibling Tools/*.swift. Mutating tools (red) pass the full gate stack and fail closed without a daemon; read-only tools (green) can fall back in-process.

Perception (read-only)

Interaction (mutating)

Apps, windows & navigation (mutating)

Exact mutating set: click, type_text, set_value, press_key, scroll, drag, select_text, perform_secondary_action, write_clipboard, open_app, open_url, manage_window — see ToolKit/ToolSpec.swift (mutatingToolNames).

11Source map — where to look

Everything is under Sources/computer-use-mcp/. Four layers:

Plus: Sources/ComputerUseFixture/ — a deterministic AppKit GUI fixture with stable AX identifiers, for the end-to-end "truth suite"; Tests/ComputerUseMCPTests/; scripts/ (live/CI eval harnesses like live_background_eval.py); docs/ (the contracts below); packaging/homebrew/.

12The contracts — read these next

This repo is unusually spec-driven: the behaviour is written down as contracts that tests, tool docs, and evals all reference. If you read nothing else after this page, read these two.

13Mental model recap

1. One binary, three roles. N thin serve shims → one daemon engine → one overlay cursor. The daemon owning all shared state is why concurrent agents can't collide.
2. One funnel. Every call — from any entry point — passes the same gates in dispatchTool. Uniform guarantees come from that single chokepoint.
3. See below the pixels. AX tree for semantics + ScreenCaptureKit for one window + ids backed by validated live AX handles. Same objects retain ids; recreated objects fail stale.
4. Act AX-first, event-last, never auto-escalate. The ladder protects your cursor and focus by default; global takeover is always an explicit opt-in.
5. Never lie. Background delivery has no OS success signal, so the server verifies the effect and reports a four-value verdict + focus telemetry instead of claiming success.
6. Yield to the human. Interference guard, screen-lock pause, URL policy, confirmation gating — all server-side, all recoverable errors, none the agent can casually disable.