# Ghost OS v2 - MCP Agent Instructions

You have Ghost OS, a tool that lets you see and operate any macOS application
through the accessibility tree AND visual perception. Every button, text field,
link, and label is available -- either through the AX tree (native apps) or
vision-based grounding (web apps where Chrome exposes everything as AXGroup).

You have 29 tools: perceive the screen, click, type, hover, drag, long-press,
scroll, press keys, manage windows, wait for conditions, run saved recipes,
and learn new workflows by watching the user.

## Rule 1: Always Check Recipes First

Before doing ANY multi-step task manually, call `ghost_recipes`.

If a recipe exists for what you need, use `ghost_run` with the recipe name
and parameters. Recipes are tested, reliable, and faster than manual steps.

## Rule 2: Orient Before Acting

Before interacting with any app, call `ghost_context` with the app name.

This tells you: which app/window is active, the current URL (for browsers),
what element is focused, and what interactive elements are visible.

**If you skip this, you will click the wrong thing.**

## Rule 3: How to Find Elements

Use `ghost_find` with the most specific identifier available:
- `dom_id` for web apps (most reliable, bypasses depth limits)
- `identifier` for native apps with developer IDs
- `query` + `role` for general searches (e.g., query:"Compose", role:"AXButton")
- `query` alone as a fallback

Use `ghost_inspect` to examine an element before acting on it.
Use `ghost_element_at` to identify elements from screenshots.

## Rule 4: How Focus Works

**Perception tools work from background** (no focus needed):
- ghost_context, ghost_state, ghost_find, ghost_read, ghost_inspect,
  ghost_element_at, ghost_screenshot

Note: `ghost_screenshot` captures windows even when they are behind other
windows or in another Space. It does NOT need the app to be focused. If all of
an app's windows are closed (not just minimized), screenshot will return an
error -- you cannot capture what does not exist.

**Click and type try AX-native first** (no focus), then synthetic fallback (auto-focuses):
- ghost_click, ghost_type

**Press, hotkey, scroll, hover, long_press, drag need focus** - always pass the `app` parameter:
- ghost_press, ghost_hotkey, ghost_scroll, ghost_hover, ghost_long_press, ghost_drag

Focus is automatically saved and restored after click and type. Other action tools leave the target app focused.

## Rule 5: Key Patterns

### Navigate Chrome to a URL
```
ghost_hotkey keys:["cmd","l"] app:"Chrome"  -> address bar focused
ghost_type text:"https://example.com"       -> URL entered
ghost_press key:"return" app:"Chrome"       -> navigate
ghost_wait condition:"urlContains" value:"example.com" app:"Chrome"
```

### Fill a form
```
ghost_click query:"Compose" app:"Chrome"    -> click button
ghost_type text:"hello@example.com" into:"To" app:"Chrome"
ghost_press key:"tab" app:"Chrome"          -> move to next field
ghost_type text:"Subject line" into:"Subject" app:"Chrome"
```

### Annotate for visual orientation
```
ghost_annotate app:"Chrome"
-> Returns labeled screenshot + text index
-> [1] Button "Send" — click: (620, 350)
-> Use ghost_click x:620 y:350 to click any labeled element
```

### Hover to reveal tooltips or menus
```
ghost_hover query:"Help" app:"Preview"
ghost_hover x:500 y:300 app:"Chrome"
```

### Drag files, sliders, or list items
```
ghost_drag query:"document.pdf" to_x:500 to_y:300 app:"Finder"
ghost_drag from_x:200 from_y:400 to_x:600 to_y:400 app:"Finder"
```

### Wait instead of guessing
```
ghost_wait condition:"elementExists" value:"Send" app:"Chrome"
ghost_wait condition:"urlContains" value:"inbox" timeout:15 app:"Chrome"
ghost_wait condition:"elementGone" value:"Loading" app:"Chrome"
```

## Rule 6: Vision Fallback for Web Apps

When `ghost_find` or `ghost_click` can't locate an element (common in web apps
like Gmail, Slack, etc. where Chrome exposes everything as AXGroup), Ghost OS
automatically falls back to VLM-based vision grounding if the vision sidecar
is running.

You can also use vision tools directly:

### ghost_ground - Find element by visual description
```
ghost_ground description:"Compose button" app:"Chrome"
-> Returns: {x: 86, y: 223, confidence: 0.8, method: "full-screen"}
```

For overlapping UI panels (e.g., compose popup over inbox), use crop_box
to narrow the search area for dramatically better accuracy:
```
ghost_ground description:"Send button" app:"Chrome" crop_box:[510, 168, 840, 390]
-> Returns: {x: 620, y: 350, confidence: 0.95, method: "crop-based"}
```

Then click at the returned coordinates:
```
ghost_click x:86 y:223 app:"Chrome"
```

### ghost_parse_screen - Detect all interactive elements
```
ghost_parse_screen app:"Chrome"
-> Returns list of all detected UI elements with bounding boxes
```
Note: Full YOLO detection is not yet implemented. Use ghost_find for
AX-based element search, and ghost_ground for visual element finding.

## Rule 7: Handle Failures

If an action fails:
1. Call `ghost_context` to see current state
2. Call `ghost_annotate` for a labeled screenshot showing every interactive element with click coordinates
3. Call `ghost_screenshot` for raw visual confirmation
4. Try `ghost_ground` with a description of what you need to click
5. Try a different approach (different query, coordinates, etc.)

Don't retry the same thing 5 times. If ghost_click fails, it already tried
AX-native, synthetic, AND VLM vision grounding. The element might not exist,
might be hidden, or might be blocked by a modal.

If `ghost_screenshot` fails for a background app, the window may be minimized
or in another Space. Ghost OS will attempt to capture it off-screen first. If
that fails, it will briefly activate the app to bring it on-screen. If the app
has no open windows at all, screenshot will return a specific error telling you.

## Rule 8: Web App Interaction (Chrome/Electron)

Chrome exposes most web elements as AXGroup -- `ghost_find` may not locate
buttons or inputs by name. **Always prefer `dom_id` for web apps.**

Pattern:
```
ghost_find query:"Send" role:AXButton app:"Chrome"  -> get dom_id from result
ghost_click dom_id:":oq" app:"Chrome"               -> click by dom_id
```

`dom_id` clicks are the MOST reliable method for any web app button.

If `ghost_find` returns nothing, use `ghost_ground` with `crop_box` for
visual grounding.

For text input in web apps, click the field first (by dom_id or coordinates),
then use `ghost_type`.

## Rule 9: Gmail / Email Pattern

Gmail's popup compose window does not reliably accept keyboard input from
synthetic events. Use URL-based compose instead:

```
Navigate to: https://mail.google.com/mail/?view=cm&fs=1&to=EMAIL&su=SUBJECT&body=BODY
```

Wait for the page to load, then find and click the Send button using dom_id.
The Send button's dom_id changes between sessions -- always `ghost_find` it
first rather than hardcoding.

## Rule 10: Coordinate Mapping

Screenshots are downsampled to 1280px max width. Pixel coordinates in the
screenshot image are NOT the same as screen coordinates.

- Use `ghost_ground` for visual-to-screen coordinate translation
- Use `ghost_find` to get element positions (always in screen coordinates)
- The screenshot response includes `window_frame` with the actual screen
  position and size of the captured window

## Rule 11: Vision Grounding Best Practices

**ALWAYS use `crop_box` with `ghost_ground`** when you know the approximate
area. It is 10x faster (250ms vs 3s) and much more accurate.

- `crop_box` format: `[x1, y1, x2, y2]` in logical screen points
- For overlapping UI (popups, dropdowns, compose windows), `crop_box` is
  ESSENTIAL to prevent the VLM from grounding to the wrong layer
- Get crop coordinates from `ghost_find` element positions or `ghost_state`
  window positions

## Rule 12: Wait Between Actions

Web apps need time to react to clicks. Always use `ghost_wait` after clicking
buttons before proceeding:

```
ghost_click query:"Submit" app:"Chrome"
ghost_wait condition:"elementExists" value:"Success" app:"Chrome"
```

Common wait conditions:
- `urlContains` -- wait for navigation
- `titleContains` -- wait for page title change
- `elementExists` -- wait for an element to appear
- `elementGone` -- wait for a loading indicator to disappear

## Rule 13: Self-Learning Mode

Ghost OS can learn workflows by watching the user perform them.

### How to use learning mode
1. User says they want to teach a workflow
2. Call `ghost_learn_start` with a task_description
3. Tell the user to perform the task
4. User says they are done
5. Call `ghost_learn_stop` to get the recorded actions
6. Analyze the actions: identify parameters (email addresses, names, URLs that should be substitutable)
7. Synthesize a recipe JSON from the actions (see Recipe JSON Schema below)
8. Verify the recipe works (see "Verifying recipes before saving" below)

### Synthesizing recipes from recordings
When you receive the action array from ghost_learn_stop:
- Each `click` with an `element` context becomes a recipe step with a target Locator
- Use `dom_id` as the primary criterion if available (most stable for web apps)
- Use `role` + `computedNameContains` as secondary criteria
- Each `typeText` where the text looks like user-specific data (email, name, URL) becomes a `{{parameter}}`
- Each `keyPress` for Tab/Return becomes a `press` step
- Each `hotkey` becomes a `hotkey` step
- Each `appSwitch` becomes a `focus` step
- Infer `wait_after` conditions from timing gaps (>2s between actions suggests a page load)
- The recipe must use schema_version 2 and be compatible with ghost_run

### Recipe JSON Schema

Every recipe saved via `ghost_recipe_save` must follow this exact structure.
Missing or misnamed fields will cause a decode error.

```json
{
  "schema_version": 2,
  "name": "my-recipe-name",
  "description": "What this recipe does",
  "app": "Google Chrome",
  "params": {
    "recipient": {
      "type": "string",
      "description": "Email address of recipient",
      "required": true
    }
  },
  "preconditions": {
    "app_running": "Google Chrome",
    "url_contains": "example.com"
  },
  "steps": [
    {
      "id": 1,
      "action": "click",
      "target": {
        "criteria": [{"attribute": "AXDOMIdentifier", "value": ":oq"}],
        "computedNameContains": "Compose"
      },
      "wait_after": {
        "condition": "elementExists",
        "value": "To recipients",
        "timeout": 5
      },
      "note": "Click the compose button"
    },
    {
      "id": 2,
      "action": "type",
      "target": {
        "criteria": [{"attribute": "AXRole", "value": "AXComboBox"}],
        "computedNameContains": "To recipients"
      },
      "params": {"text": "{{recipient}}"},
      "note": "Type recipient email"
    },
    {
      "id": 3,
      "action": "press",
      "params": {"key": "tab"},
      "note": "Confirm autocomplete and move to next field"
    },
    {
      "id": 4,
      "action": "wait",
      "params": {"condition": "elementExists", "value": "Subject", "timeout": "5"},
      "note": "Wait for subject field to be ready"
    },
    {
      "id": 5,
      "action": "hotkey",
      "params": {"keys": "cmd,return"},
      "note": "Send with Cmd+Return"
    },
    {
      "id": 6,
      "action": "wait",
      "params": {"condition": "elementGone", "value": "Send", "timeout": "10"},
      "on_failure": "skip",
      "note": "Verify send completed (optional)"
    }
  ],
  "on_failure": "stop"
}
```

**Required fields:**
- `schema_version` -- must be `2`
- `name` -- string, used as filename and recipe identifier
- `description` -- string, shown in `ghost_recipes` listing
- `steps` -- array, at least one step

**Optional top-level fields:**
- `app` -- default app for all steps
- `params` -- parameter definitions (see below)
- `preconditions` -- conditions to check before running
- `on_failure` -- `"stop"` (default) or `"skip"`. No other values are valid.

**Step fields:**
- `id` (required) -- integer, sequential starting from 1
- `action` (required) -- one of: `click`, `type`, `press`, `hotkey`, `focus`, `scroll`, `wait`, `hover`, `long_press`, `drag`
- `target` (optional) -- Locator object for finding the element to act on
- `params` (optional) -- action-specific parameters (see below). All values are **strings**.
- `wait_after` (optional) -- condition to wait for after this step completes
- `note` (optional) -- human-readable description of what this step does
- `on_failure` (optional) -- `"stop"` or `"skip"` (overrides recipe-level setting)

**Action params by type:**

| Action | Required params | Optional params |
|--------|----------------|-----------------|
| `click` | (none -- use `target`) | `x`, `y`, `button` (left/right/middle), `count` (2=double, 3=triple), `query` |
| `type` | `text` | `into`, `clear` ("true"), `app` |
| `press` | `key` | `modifiers` (comma-separated: cmd,shift,option,control), `app` |
| `hotkey` | `keys` (comma-separated: cmd,return) | `app` |
| `focus` | `app` | `window` |
| `scroll` | `direction` (up/down/left/right) | `amount`, `x`, `y`, `app` |
| `wait` | `condition` | `value`, `timeout` (seconds as string, default "10") |
| `hover` | (none -- use `target`) | `x`, `y`, `query`, `app` |
| `long_press` | (none -- use `target`) | `x`, `y`, `duration`, `button`, `query`, `app` |
| `drag` | `to_x`, `to_y` | `from_x`, `from_y`, `duration`, `hold_duration`, `query`, `app` |

Key names for `press`: return, tab, escape, space, delete, up, down, left, right, f1-f12.

**Important:** The `wait` action is different from `wait_after`. A `wait` step is
a standalone polling action with params like `{"condition": "elementExists", "value": "...", "timeout": "5"}`.
Note that `timeout` is a **string** (all step params are strings). `wait_after` is a
structured object on any step with `timeout` as a **number**. See the bundled
`slack-send` recipe for an example using both.

**Target (Locator) fields:**
- `criteria` -- array of `{"attribute": "...", "value": "..."}` objects. Common attributes:
  - `AXRole` -- element type (AXButton, AXTextField, AXTextArea, AXComboBox, AXLink, AXStaticText)
  - `AXDOMIdentifier` -- DOM id for web apps (most reliable for Chrome/Electron)
- `computedNameContains` -- string, matches element name/title/description (most useful field)
- `matchAll` -- boolean, whether all criteria must match (default: true)

**wait_after fields:**
- `condition` -- one of: `elementExists`, `elementGone`, `urlContains`, `titleContains`, `urlChanged`, `titleChanged`, `delay`
- `value` -- string to match against (not needed for `delay`, `urlChanged`, or `titleChanged`)
- `target` -- optional Locator; if `value` is nil, falls back to `target.computedNameContains`
- `timeout` -- number (seconds, default: 10). For `delay`, this is the sleep duration.

The same conditions work for standalone `wait` steps (pass as `params.condition`).

**Param definitions** (top-level `params` object):
- Each key is the parameter name
- `type` (required) -- `"string"`
- `description` (required) -- what this parameter is for
- `required` (optional) -- boolean

Use `{{param_name}}` in step params to substitute recipe parameters at runtime.

### Verifying recipes before saving

**Always test a recipe before considering it done.** This applies whether the recipe
came from self-learning mode or was authored from scratch.

1. Save the recipe with `ghost_recipe_save`
2. Ask the user for safe test parameters (especially for recipes that send messages,
   emails, or modify files -- you don't want to send a real email during testing)
3. Run with `ghost_run` using the test parameters
4. If the run fails: `ghost_recipe_delete`, fix the JSON, go back to step 1
5. Repeat until the recipe runs successfully end-to-end
6. Tell the user the recipe is verified and ready

**Do not skip verification.** A recipe that fails on first use is worse than no
recipe at all.

### Requirements
- Input Monitoring permission required (separate from Accessibility)
- Only records between ghost_learn_start and ghost_learn_stop (no background monitoring)
- Password fields are automatically redacted
- Recordings are ephemeral (in memory only, never written to disk)
- Do not call ghost_run while ghost_learn is active (synthetic events would be re-recorded)

## Tool Reference

| Tool | Purpose | Needs Focus? |
|------|---------|-------------|
| ghost_context | Where am I? URL, focused element, actions | No |
| ghost_state | All running apps and windows | No |
| ghost_find | Find elements by text, role, DOM id | No |
| ghost_read | Read text content from screen | No |
| ghost_inspect | Full element metadata | No |
| ghost_element_at | What's at these coordinates? | No |
| ghost_screenshot | Visual capture for debugging | No |
| ghost_annotate | Labeled screenshot with click coordinates | No |
| ghost_click | Click element or coordinates | Auto |
| ghost_type | Type text, optionally into a field | Auto |
| ghost_hover | Move cursor to trigger tooltips/hover effects | Yes - use `app` |
| ghost_long_press | Press and hold for context menus | Yes - use `app` |
| ghost_drag | Drag between points or elements | Yes - use `app` |
| ghost_press | Press single key | Yes - use `app` |
| ghost_hotkey | Key combo (cmd+s, etc.) | Yes - use `app` |
| ghost_scroll | Scroll content | Yes - use `app` |
| ghost_focus | Bring app to front | N/A |
| ghost_window | Window management | No |
| ghost_wait | Wait for condition | No |
| ghost_recipes | List recipes | No |
| ghost_run | Execute recipe | Auto |
| ghost_recipe_show | View recipe details | No |
| ghost_recipe_save | Save new recipe | No |
| ghost_recipe_delete | Delete recipe | No |
| ghost_parse_screen | Detect ALL UI elements via vision | No |
| ghost_ground | Find element coordinates via VLM | No |
| ghost_learn_start | Start observing user actions for learning | No |
| ghost_learn_stop | Stop observing and return recorded actions | No |
| ghost_learn_status | Check learning mode status | No |
