Desktop Control Default Active

Computer Use desktop automation — get_app_state, click, set_value, select_text, keyboard/pointer actions. Unified skill for all UI automation that routes between CDP (browser DOM) and Computer Use (desktop apps) based on the target.

Quick Reference

Execution Paths

Action Classes

Intent Routing

The skill decides the execution path before acting. The first line of every task must announce path=cdp, path=computer-use, or path=cdp+cu.

Decision Table

Resolution Order

1. Explicit opt-in: Message contains $computer-use or /computer-use → Computer Use path immediately. If tools unavailable, stop with precondition failed: computer-use unavailable.
2. Can the target be addressed by cli-jaw browser snapshot --interactive ref? → CDP.
3. Non-DOM web widget (Canvas, WebGL, iframe) visible in get_app_state screenshot? → Computer Use click(x, y).
4. Target outside any webpage (app window, menu bar, OS dialog)? → Computer Use.
5. Pixel coordinate the user gave verbatim? → Computer Use pointer-action.
6. Need DOM to locate but user insists on real cursor click? → Hybrid.
7. App name unclear? → list_apps() before get_app_state(app).
8. None matched → stop and report needs boss follow-up: ambiguous target.

Absolute Rules

These rules are non-negotiable. Violating any of them produces incorrect transcripts or wasted actions.

1. Announce the path before acting. First line of every task must be path=cdp, path=computer-use, or path=cdp+cu.
2. State-first rule. Computer Use always starts each assistant turn with get_app_state(app) before interacting. Re-call it on stale warnings, after UI-changing actions, and whenever confidence drops.
3. Every meaningful action records an action_class. Classes: state-read, element-action, value-injection, keyboard-action, pointer-action, pointer-action+vision.
4. Never fall back silently. If the required path is unavailable, stop and report which precondition failed.
5. Never claim the cursor was visible. Cursor overlay is best-effort in the current build.
6. When uncertain, take a screenshot FIRST. Never chain actions through uncertainty. If two consecutive actions produced ambiguous state, the next call must be a state-read.

Computer Use Tool Surface

The Computer Use path exposes these tools through the active runtime:

CDP Path — cli-jaw browser

For anything addressable by Chrome DOM: page text, form fields, buttons inside the page. 10x faster than Computer Use for web content.

Session Commands

cli-jaw browser status                         # is a session active?
cli-jaw browser start --agent                  # headless automation session
cli-jaw browser start --headless               # manual headless (WSL/CI/Docker)
cli-jaw browser start                          # visible window (only if user asks)
cli-jaw browser navigate "https://example.com" # go to URL
cli-jaw browser open "https://example.com"     # open in new tab
cli-jaw browser tabs                           # list tabs

Snapshot + Act

cli-jaw browser snapshot --interactive   # list clickable elements (e1, e2, ...)
cli-jaw browser click e3                 # click by ref
cli-jaw browser type e5 "query" --submit # type + Enter
cli-jaw browser press Enter              # fallback key press
cli-jaw browser screenshot               # save a PNG
cli-jaw browser text                     # dump visible text

CDP Action Class Mapping

Preconditions

Computer Use Path

• Platform: macOS only.
• Active agent runtime must expose all Computer Use tools (list_apps, get_app_state, click, drag, press_key, scroll, select_text, set_value, type_text, perform_secondary_action).
• TCC Accessibility and AppleEvents must be granted to the controlling app.
• In packaged installs, /Applications/Jaw.app and /Applications/Codex Computer Use.app may be required for TCC attribution.
• If app name is unknown, call list_apps() first.

CDP Path

• cli-jaw serve must be running (check with cli-jaw browser status).
• A browser session must be active (start with cli-jaw browser start --agent).

Transcript Format

Every action is logged in a structured transcript. The boss parses these — do not omit the path= line.

CDP Transcript

path=cdp
url=https://example.com
action=click e3
result=ok

Computer Use Transcript

path=computer-use
app=Google Chrome
action_class=element-action
action=click(element_index=730)
stale_warning=no
result=ok

Hybrid Transcript

path=cdp+cu
lookup=cli-jaw browser snapshot -> bbox of "Play"
action_class=pointer-action
action=click(x=812, y=514)
result=ok

Error Transcript

path=computer-use
app=Finder
action_class=element-action
action=click(element_index=34)
stale_warning=yes
result=error: stale element tree

Stale Recovery Pattern

When stale_warning=yes is returned, the element tree has changed. Follow this exact sequence:

1. get_app_state(app) — re-ground; note the real element_index of your target.
2. Log action_class=stale-recovery (CU-06) with a one-line note (reason=disambiguation).
3. Re-issue the intended action using the fresh element_index.
4. After that action, get_app_state(app) once more to confirm the effect.

# Stale warning received
path=computer-use
app=Google Chrome
action_class=stale-recovery
action=get_app_state("Google Chrome")
stale_warning=no
result=ok (52 elements -- tree changed)

# Retry with fresh index
path=computer-use
app=Google Chrome
action_class=element-action
action=click(element_index=38)   # same target, new index
stale_warning=no
result=ok

Vision-Click Integration

Vision-click is one specific tactic inside the broader UI targeting problem. It is legacy and generally superseded by direct coordinate clicking from screenshots.

Decision Order

1. Did cli-jaw browser snapshot --interactive return a ref? → CDP click.
2. Is Computer Use available and target visible in screenshot? → click(x, y) pointer-action directly (preferred).
3. No ref and direct coordinate unsuitable → cli-jaw browser vision-click (Codex-only, legacy).

# CDP vision-click (inside Chrome, Codex-only)
cli-jaw browser vision-click "Submit button"
cli-jaw browser vision-click "Play button" --double

# Desktop vision-click (CU-05)
# 1. get_app_state(app) to capture screenshot
# 2. Vision model identifies coordinates
# 3. click(x=vx, y=vy) via Computer Use

Worked Example: Chrome → Spotify Trace

A real end-to-end trace demonstrating state-first, element targeting, stale recovery, and the CDP speed switch.

1. State First

path=computer-use
app=Google Chrome
action_class=state-read
action=get_app_state("Google Chrome")
stale_warning=no
result=ok (47 elements, focused tab: open.spotify.com)

2. Element-Index Targeting (not focus-only typing)

# Click the search input field
path=computer-use
app=Google Chrome
action_class=element-action
action=click(element_index=12)
stale_warning=no
result=ok

# Set search value deterministically
path=computer-use
app=Google Chrome
action_class=value-injection
action=set_value(element_index=12, value="Daft Punk")
stale_warning=no
result=ok

3. Stale Warning → Recovery

# Search results load asynchronously -- tree changes
path=computer-use
app=Google Chrome
action_class=element-action
action=click(element_index=34)
stale_warning=yes
result=error: stale element tree

# Correct recovery
path=computer-use
app=Google Chrome
action_class=stale-recovery
action=get_app_state("Google Chrome")
result=ok (52 elements -- tree changed)

path=computer-use
app=Google Chrome
action_class=element-action
action=click(element_index=38)   # same result, new index
stale_warning=no
result=ok

4. Switch to CDP for 10x Speed

# Realize all targets are DOM nodes -- swap to CDP
path=cdp
url=https://open.spotify.com
action=cli-jaw browser snapshot --interactive
result=ok (ref IDs: e1..e89)

path=cdp
url=https://open.spotify.com
action=click e42   # "Shuffle Play" button
result=ok

Common Failures & Correct Responses

Contract ID Reference

All routing maps to capability contracts defined in the internal Computer Use spec. Use these IDs in transcripts when a contract name is needed.

"~해줘" Usage Examples

Natural Korean requests that trigger the desktop-control skill. The routing system understands both Korean and English.

Rules: Never Do

• Do not try the wrong path once "just in case" — it burns TCC prompts and confuses logs.
• Do not silently upgrade from state-read to element-action without updating the action class.
• Do not omit the path= line. The boss parses it.
• Do not use curl/wget to "read a page" — always use CDP.
• Do not open a visible browser window just to inspect logs — use the Web UI debug console.
• Do not try coordinate-based click when a ref exists — use the ref.
• Do not assume the cursor is visible. Report action success/failure only.
• Do not skip get_app_state because "you remember where the button is." Element indices drift with every state change.
• Do not run two Computer Use actions without re-reading state between them if the previous action changed anything.
• Do not resolve uncertainty by trial and error. "Let me click 342, if not, try 357" is forbidden — take a screenshot.
• Do not use type_text(app, text) as a shortcut for targeted form entry unless focus was verified in the latest state.
• Do not retry vision-click with rephrasings — one attempt per call; report and stop on failure.

Related Skills