# Smithers Observability

> Smithers observability surface: HTTP server, gateway, MCP, OpenTelemetry, metrics.

---

## HTTP Server

> Run Smithers as an HTTP server: REST routes for runs, approvals, tools.

`startServer` boots a multi-workflow HTTP server with REST endpoints for run lifecycle, SSE event streams, and [human-in-the-loop approvals](/components/approval). For a single-workflow variant alongside `bunx smithers-orchestrator up`, see [Serve Mode](/integrations/serve).

<Note>API reference: [Server & Gateway](/reference/server-gateway) lists every server and gateway export, its options, and links to source and tests.</Note>

## Quick start

```ts
import { startServer } from "smithers-orchestrator";
import { drizzle } from "drizzle-orm/bun-sqlite";

const server = startServer({
  port: 7331,
  db: drizzle("./smithers.db"),
  authToken: process.env.SMITHERS_API_KEY,
  rootDir: process.cwd(),
});
```

```bash
curl -X POST http://localhost:7331/v1/runs \
  -H "Authorization: Bearer $SMITHERS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"workflowPath": "./bugfix.tsx", "input": {"description": "fix auth"}}'
```

## ServerOptions

```ts
type ServerOptions = {
  port?: number;              // default 7331
  host?: string;              // default 127.0.0.1; a non-loopback bind requires authToken
  db?: unknown;               // enables GET /v1/runs and approvals listing
  authToken?: string;         // falls back to process.env.SMITHERS_API_KEY
  insecure?: boolean;         // default false; allow a non-loopback bind with no authToken (dangerous)
  maxBodyBytes?: number;      // default 1_048_576
  rootDir?: string;           // sandbox root for workflow paths and tools
  allowNetwork?: boolean;     // default false; allows network in `bash`
  integrations?: IntegrationsConfig; // external HMAC-verified integrations served by this process (requires db)
  headersTimeout?: number;    // default 30_000
  requestTimeout?: number;    // default 60_000
};
```

`startServer` returns a listening `http.Server`. `headersTimeout` and `requestTimeout` are applied to that server to bound slow header/body uploads.

## API Routes (TOON)

```toon
routes[15]{method,path,purpose,auth}:
  GET,/metrics,Prometheus exposition,bearer
  POST,/v1/runs,Start or resume a run,bearer
  GET,/v1/runs,List runs (requires db),bearer
  GET,/v1/runs/:runId,Run status and node summary,bearer
  POST,/v1/runs/:runId/resume,Resume paused or failed run,bearer
  POST,/v1/runs/:runId/cancel,Abort an active run,bearer
  GET,/v1/runs/:runId/events,SSE event stream (?afterSeq=N),bearer
  GET,/v1/runs/:runId/frames,List render frames,bearer
  POST,/v1/runs/:runId/nodes/:nodeId/approve,Approve a paused node,bearer
  POST,/v1/runs/:runId/nodes/:nodeId/deny,Deny a paused node,bearer
  POST,/v1/runs/:runId/signals/:signalName,Deliver a named signal,bearer
  GET,/v1/approvals,List pending approvals (requires db),bearer
  GET,/v1/approval/list,Legacy alias for /v1/approvals,bearer
  GET,/approval/list and /approvals,Legacy aliases for /v1/approvals,bearer
  POST,/signal/:runId/:signalName,Legacy alias for signals,bearer
```

JSON requests/responses use `Content-Type: application/json`, `Cache-Control: no-store`, and `X-Content-Type-Options: nosniff`. SSE events are named `smithers` and carry [`SmithersEvent`](/runtime/events) JSON; the stream sends a keep-alive comment every 10 s and closes on terminal state.

Errors use the envelope `{ "error": { "code", "message", "details" } }`. Common codes: `INVALID_REQUEST`, `INVALID_JSON`, `PAYLOAD_TOO_LARGE`, `RUN_ID_REQUIRED`, `RUN_NOT_FOUND`, `RUN_ALREADY_EXISTS`, `RUN_NOT_ACTIVE`, `NOT_FOUND`, `UNAUTHORIZED`, `WORKFLOW_PATH_OUTSIDE_ROOT`, `DB_NOT_CONFIGURED`, `SERVER_ERROR`.

## Tool surface

Tools resolve relative to `rootDir`. The example below exposes a workflow that uses the built-in `bash` tool through the server; clients call it via `POST /v1/runs`.

```tsx
/** @jsxImportSource smithers-orchestrator */
import { Task, Workflow, createSmithers } from "smithers-orchestrator";
import { bashTool } from "smithers-orchestrator/tools";
import { z } from "zod";

const { smithers, outputs } = createSmithers({
  result: z.object({ stdout: z.string() }),
});

export default smithers((ctx) => (
  <Workflow name="echo">
    <Task id="run" output={outputs.result}>
      {async () => ({ stdout: await bashTool("echo", [ctx.input.msg]) })}
    </Task>
  </Workflow>
));
```

`allowNetwork: false` (the default) keeps `bash` offline. Set `rootDir` to constrain the filesystem the workflow can touch.

## Authentication

When `authToken` is configured (directly or via `SMITHERS_API_KEY`), every request must include either:

- `Authorization: Bearer <token>`, or
- `x-smithers-key: <token>`.

Missing or invalid tokens return `401`. No scopes; for finer access control use the [Gateway](/integrations/gateway).

## Notes

- Each `POST /v1/runs` and `/resume` reloads the workflow source via a content-addressed shadow file with the same extension as the source, so edits take effect on the next call without a restart.
- Active runs heartbeat to `_smithers_runs.heartbeat_at_ms` every 1 s; stale rows (no heartbeat in 30 s) are treated as crashed and may be resumed.
- When a server-level `db` differs from a workflow's database, runs and events are mirrored asynchronously to the server db so they show up in `GET /v1/runs`.
- Metrics are exported via `/metrics`; set `SMITHERS_OTEL_ENABLED=1` plus `OTEL_EXPORTER_OTLP_ENDPOINT` for OTLP. See [Observability](/guides/monitoring-logs).

---

## Serve Mode

> Run a single workflow as an HTTP server with Hono. Interact with it over REST, stream events via SSE, and manage approvals remotely.

Serve mode starts a Hono-based HTTP server alongside a running workflow. Every route operates on the single active run, with no workflow path or run ID in requests.

<Note>API reference: [Server & Gateway](/reference/server-gateway) lists every server and gateway export, its options, and links to source and tests.</Note>

## CLI

```bash
bunx smithers-orchestrator up workflow.tsx --serve --port 3000 --host 0.0.0.0 --auth-token "$SMITHERS_API_KEY"
```

A non-loopback `--host` (like `0.0.0.0`) requires `--auth-token` (or `--insecure` to opt out, which is dangerous). Binding to the default `127.0.0.1` needs neither.

| Flag | Default | Description |
|---|---|---|
| `--serve` | `false` | Enable HTTP server mode |
| `--port` | `7331` | TCP port |
| `--host` | `127.0.0.1` | Bind address |
| `--auth-token` | `SMITHERS_API_KEY` env | Bearer token for auth; required for a non-loopback `--host` |
| `--insecure` | `false` | Allow a non-loopback `--host` with no auth (exposes unauthenticated approve/deny/cancel; dangerous) |
| `--metrics` | `true` | Expose `/metrics` Prometheus endpoint |
| `--supervise` | `false` | Run the stale-run supervisor loop alongside `--serve` |

The process stays alive after the workflow completes so final state remains queryable. Ctrl+C stops both the server and the workflow.

Detached mode:

```bash
bunx smithers-orchestrator up workflow.tsx --serve --port 8080 -d
```

## Programmatic

```ts
import { SmithersDb, createServeApp } from "smithers-orchestrator";

const adapter = new SmithersDb(workflow.db);
const abort = new AbortController();

const app = createServeApp({
  workflow,
  adapter,
  runId,
  abort,
  authToken: "sk-secret",
});

Bun.serve({ port: 3000, fetch: app.fetch });
```

`createServeApp` returns a standard Hono app. Mount it with `Bun.serve`, pass it to another Hono app via `app.route()`, or use `app.fetch` in tests.

## ServeOptions

```ts
type ServeOptions = {
  workflow: SmithersWorkflow<unknown>;   // loaded workflow instance
  adapter: SmithersDb;               // Smithers DB adapter; e.g. new SmithersDb(workflow.db)
  runId: string;                     // active run ID
  abort: AbortController;            // shared cancellation controller
  authToken?: string;                // bearer token; falls back to SMITHERS_API_KEY; disabled if unset
  metrics?: boolean;                 // expose /metrics; default true
};
```

---

## Authentication

When `authToken` is configured, every route except `/health` requires:

- `Authorization: Bearer <token>`, or
- `x-smithers-key: <token>`

Missing or invalid tokens receive `401`.

---

## Routes

### GET /health

Returns `200` regardless of auth.

```json
{ "ok": true }
```

### GET /

Run status and node summary.

```json
{
  "runId": "run-1234",
  "workflowName": "bugfix",
  "status": "running",
  "startedAtMs": 1707500000000,
  "finishedAtMs": null,
  "summary": { "finished": 3, "in-progress": 1, "pending": 2 }
}
```

### GET /events

SSE stream of lifecycle events. Same format as the [multi-workflow server](/integrations/server#api-routes-toon).

| Parameter | Type | Default | Description |
|---|---|---|---|
| `afterSeq` | `number` | `-1` | Only events after this sequence |

```
event: smithers
data: {"type":"NodeStarted","runId":"run-1234","nodeId":"analyze","iteration":0,"attempt":1}
id: 1

event: smithers
data: {"type":"NodeFinished","runId":"run-1234","nodeId":"analyze","iteration":0,"attempt":1}
id: 2
```

- Polls every 500ms.
- Auto-closes when the run reaches a terminal state.
- Reconnect with `?afterSeq=N` to resume from a known position.

### GET /frames

Rendered workflow frames.

| Parameter | Type | Default | Description |
|---|---|---|---|
| `limit` | `number` | `50` | Max frames |
| `afterFrameNo` | `number` | - | Frames after this number |

### POST /approve/:nodeId

Approve a pending approval gate. All fields optional. Returns `{ "runId": "run-1234" }`.

```json
{
  "iteration": 0,
  "note": "Looks good",
  "decidedBy": "alice"
}
```

### POST /deny/:nodeId

Deny a pending approval gate. Same body as `/approve/:nodeId`.

### POST /cancel

Cancel the running workflow.

| Status | Code | Condition |
|---|---|---|
| 200 | - | Cancelled successfully |
| 409 | `RUN_NOT_ACTIVE` | Run is not actively running (e.g. already finished, failed, cancelled, continued, or its heartbeat has gone stale) |

> **Note:** Runs in `waiting-approval` or `waiting-timer` state are cancelled immediately and return `200`.

### GET /metrics

Prometheus text exposition. Same metrics as the [multi-workflow server](/integrations/server#api-routes-toon).

---

## Error Format

```json
{
  "error": {
    "code": "ERROR_CODE",
    "message": "Human-readable description"
  }
}
```

Unknown routes return `404` with code `NOT_FOUND`.

---

## Serve Mode vs Multi-Workflow Server

| | Serve mode | [Multi-workflow server](/integrations/server) |
|---|---|---|
| Scope | Single workflow, single run | Any workflow, multiple concurrent runs |
| Start | `bunx smithers-orchestrator up --serve` or `createServeApp()` | `startServer()` |
| Routes | `/`, `/events`, `/approve/:nodeId`, ... | `/v1/runs`, `/v1/runs/:runId`, ... |
| Framework | Hono | Node.js `http` |
| Use case | Development, single-purpose services | Production API gateway |

---

## Example

```bash
# Start a workflow with serve mode
bunx smithers-orchestrator up workflow.tsx --serve --port 8080 --auth-token sk-secret

# Status
curl http://localhost:8080/ -H "Authorization: Bearer sk-secret"

# Stream events
curl -N http://localhost:8080/events -H "Authorization: Bearer sk-secret"

# Approve
curl -X POST http://localhost:8080/approve/deploy \
  -H "Authorization: Bearer sk-secret" \
  -H "Content-Type: application/json" \
  -d '{"note": "Ship it", "decidedBy": "alice"}'

# Health (no auth)
curl http://localhost:8080/health
```

---

## Gateway

> WebSocket / RPC gateway for connecting clients and custom UIs to Smithers runs, providing pushed updates, subscriptions, metrics, and resilient reconnection.

`Gateway` is Smithers' headless control plane. Reach for it (instead of [`startServer()`](/integrations/server)) when long-lived clients (bots, dashboards, schedulers, and custom UIs) need to authenticate once, stream events over WebSocket with resilient reconnection, decide [approvals](/components/approval), inject [signals](/runtime/events), access metrics, and manage cron schedules across many registered workflows. Custom UIs, whether using the vanilla SDK or React hooks, rely on the Gateway to provide pushed updates and a stale-data-free model. For the single-workflow Hono-based HTTP surface, see [Serve Mode](/integrations/serve) (`createServeApp()` / `bunx smithers-orchestrator up --serve`).

<Warning>
  **Gateway-first client contract:** controllers, Bun cron jobs, monitors, bots,
  and custom clients must use this Gateway API. Do not open `smithers.db`,
  `.smithers/pg`, or Postgres directly; do not query `_smithers_*`; do not import
  `openSmithersStore` or CLI-internal `findAndOpenDb`; and do not pass
  `--backend` to read/control commands to hunt for runs. Direct backend access
  exists for runtime implementation, `smithers migrate`, backend integration
  tests, and maintainer diagnostics only. It is not a control-plane API.
</Warning>

<Note>API reference: [Server & Gateway](/reference/server-gateway) and [Gateway Client](/reference/gateway-client) list every gateway and client export, its options, and links to source and tests.</Note>

## Quick start

```tsx
/** @jsxImportSource smithers-orchestrator */
import { Gateway, Task, Workflow, createSmithers } from "smithers-orchestrator";
import { z } from "zod";

const { smithers, outputs } = createSmithers({
  result: z.object({ ok: z.boolean() }),
});

const deploy = smithers((ctx) => (
  <Workflow name="deploy">
    <Task id="ship" output={outputs.result}>{{ ok: true }}</Task>
  </Workflow>
));

const gateway = new Gateway({
  heartbeatMs: 15_000,
  auth: {
    mode: "token",
    tokens: { "operator-token": { role: "operator", scopes: ["*"] } },
  },
});

gateway.register("deploy", deploy, { schedule: "0 8 * * 1-5" });
await gateway.listen({ port: 7331 });
```

```ts
const ws = new WebSocket("ws://localhost:7331");
ws.onmessage = (m) => console.log(JSON.parse(m.data));
ws.onopen = () => ws.send(JSON.stringify({
  type: "req",
  id: "c1",
  method: "connect",
  params: {
    minProtocol: 1,
    maxProtocol: 1,
    client: { id: "docs-example", version: "1.0.0", platform: "browser" },
    auth: { token: "operator-token" },
  },
}));
```

## Gateway client SDK

Programmatic clients (bots, schedulers, dashboards, third-party UIs) talk to the Gateway through the typed client SDK over the same RPC and WebSocket API. For the full custom-UI guide (declarative queries, pushed updates, stale guards, reconnect/resume, backpressure, optimistic mutations, auth, vanilla JS + React hooks) see [Custom UIs](/integrations/custom-ui).

```ts
import { SmithersGatewayClient } from "smithers-orchestrator/gateway-client";

const gateway = new SmithersGatewayClient();
const workflows = await gateway.listWorkflows();

// Resilient pushed updates: backoff + jitter on reconnect, resume from the last
// observed seq via run.gap_resync, stop on run.completed or abort.
for await (const frame of gateway.streamRunEventsResilient({ runId: "run-1" })) {
  if (frame.event === "run.completed") break;
}
```

Gateway client exports:

| Package | Exports |
|---|---|
| `smithers-orchestrator/gateway-client` | `SmithersGatewayClient`, `SmithersGatewayConnection`, `GatewayRpcError`, `gatewayBackoffDelay`, RPC frame/type-map types, extension envelope helpers/types, `GatewayUiBootConfig`, `SmithersGatewayClientOptions`, `createSmithersCollections`, `createSmithersDataClient`, `WorkspaceMode`, `SmithersCollections`, `SmithersDataClient`, collection row types, `gatewayKeys`, `smithersCollectionKeys`, `smithersLocalCollectionOptions`, `smithersElectricCollectionOptions`, `flattenGatewayRunNode`, `snapshotToGatewayRunNode`, `reconcileSnapshotNodes` |
| `smithers-orchestrator/gateway-react` | `SmithersGatewayProvider`, `SmithersCollectionsProvider`, `createGatewayReactRoot`, `useSmithersCollections`, `useGatewayRun`, `useGatewayRuns`, `useGatewayWorkflows`, `useGatewayApprovals`, `useGatewayNodeOutput`, `useGatewayRunEvents`, `useGatewayActions`, `useGatewayRpc`, `useSmithersGateway`, `useGatewayExtensionResource`, `useGatewayExtensionAction`, `useGatewayExtensionStream`, `useGatewayRunTree`, `useGatewayConnectionStatus`, `useGatewayCrons`, `useGatewayMemoryFacts`, `useGatewayScores`, `useGatewayTickets`, `useGatewayPrompts` |

## Bun controllers and cron jobs

Run one Gateway singleton per workspace. Discover its verified URL with the
public `gateway status` command rather than reading the runtime state file or
assuming port 7331 (the preferred port may already be occupied):

```ts
import { SmithersGatewayClient } from "smithers-orchestrator/gateway-client";

const statusProcess = Bun.spawnSync([
  "bunx", "smithers-orchestrator", "gateway", "status", "--format", "json",
], {
  cwd: process.cwd(),
  stdout: "pipe",
  stderr: "pipe",
});

if (!statusProcess.success) {
  throw new Error(new TextDecoder().decode(statusProcess.stderr));
}

const status = JSON.parse(new TextDecoder().decode(statusProcess.stdout));
if (!status.running || typeof status.url !== "string") {
  throw new Error("No healthy workspace Gateway is running");
}

const gateway = new SmithersGatewayClient({
  baseUrl: status.url,
  token: process.env.SMITHERS_API_KEY,
  client: { id: "local-health-cron", platform: "bun" },
});

const runs = await gateway.listRuns({ filter: { limit: 50 } });
```

Start `bunx smithers-orchestrator gateway` under your service manager when
`status.running` is false. A health cron should report the missing Gateway and
restart that singleton; it should not fall back to opening a store. Once it has
a run ID, use `getRun` for snapshots and `streamRunEventsResilient` for a live
cursor. Use `launchRun`, `resumeRun`, `cancelRun`, `submitApproval`, and
`submitSignal` for mutations so authentication, ownership, idempotency,
invalidation, and event emission stay intact.

<Note>
  **Bundling in Next.js / webpack:** in published versions up to and including
  0.28.0, `gateway-client` transitively imports
  `@smithers-orchestrator/electric-proxy`, which shipped raw TypeScript
  sources. Bun and Vite handle that natively, but webpack-based builds
  (Next.js) fail hard unless you add it to
  [`transpilePackages`](https://nextjs.org/docs/app/api-reference/config/next-config-js/transpilePackages):

  ```js
  // next.config.mjs
  export default {
    transpilePackages: ["@smithers-orchestrator/electric-proxy"],
  };
  ```

  Releases after 0.28.0 ship the package as plain JavaScript with bundled type
  declarations, so the workaround is not needed there.
</Note>

## REST domain API

The Gateway also exposes a typed REST domain surface under `/v1/api/*`. These
routes share the same auth modes, scope checks, validation, and backend code
paths as their `/v1/rpc/<method>` twins. Responses use `{ ok: true, data }` for
reads and `{ ok: false, error }` for failures.

Mutating routes include a write acknowledgement. SQLite and embedded PGlite
return `{ seq }`, where `seq` is the invalidation frame emitted by
`GET /v1/api/stream`. Real Postgres returns `{ txid }`, a numeric string from
`pg_current_xact_id()::xid::text`, for Electric transaction matching.

```toon
api[24]{route,rpc,scope}:
  GET /v1/api/runs,listRuns,run:read
  GET /v1/api/runs/:id,getRun,run:read
  POST /v1/api/runs,launchRun,run:write
  POST /v1/api/runs/:id/resume,resumeRun,run:write
  POST /v1/api/runs/:id/cancel,cancelRun,run:write
  POST /v1/api/runs/:id/rewind,rewindRun,run:admin
  GET /v1/api/runs/:id/tree,getDevToolsSnapshot,observability:read
  GET /v1/api/events,streamRunEvents tail snapshot,run:read
  GET /v1/api/nodes/:runId/:nodeId/output,getNodeOutput,run:read
  GET /v1/api/nodes/:runId/:nodeId/diff,getNodeDiff,run:read
  GET /v1/api/approvals,listApprovals,run:read
  POST /v1/api/approvals/:id,submitApproval,approval:submit
  POST /v1/api/signals,submitSignal,signal:submit
  GET /v1/api/workflows,listWorkflows,run:read
  GET /v1/api/docs,listDocs,run:read
  GET /v1/api/prompts,listPrompts,prompt:read
  GET /v1/api/scores,listScores,score:read
  GET /v1/api/tickets,listTickets,ticket:read
  GET /v1/api/memory-facts,listMemoryFacts,memory:read
  GET /v1/api/crons,cronList,cron:read
  GET /v1/api/accounts,listAccounts,account:read
  GET /v1/api/schema-signature,getSchemaSignature,run:read
  GET /v1/api/stream,SSE invalidation feed,run:read
```

`GET /v1/api/stream` is a `text/event-stream` invalidation feed for local
TanStack Query collections. Change events are `event: change` frames with
`data: {"seq": number, "collections": string[]}`. Writes that happen inside a
short coalescing window emit one frame with the union of collection names.

Clients should send `Last-Event-ID` when reconnecting. The Gateway replays
missed frames from its bounded ring buffer when possible. If the requested seq
is too old, it sends `event: reset` with the current seq so the client can
invalidate all collection query keys. Each connection also has a bounded
outbound queue; a slow consumer is resynced through the same reset path, which
bounds memory per stream.

## RPC methods (TOON)

```toon
rpc[27]{method,params,returns,scope,transport}:
  launchRun,workflow/input?/options.runId?/options.idempotencyKey?,{runId/workflow},run:write,http+websocket
  resumeRun,runId/options.force?,{runId/status},run:write,http+websocket
  cancelRun,runId,{runId/status:cancelling},run:write,http+websocket
  hijackRun,runId/options?,{runId/status:hijack-ready/sessionId},run:admin,http+websocket
  rewindRun,runId/frameNo/confirm:true,JumpResult,run:admin,http+websocket
  submitApproval,runId/nodeId/iteration?/decision,{runId/nodeId/iteration/approved},approval:submit,http+websocket
  submitSignal,runId/correlationKey/payload?/signalName?,Delivery metadata,signal:submit,http+websocket
  getRun,runId,Run record + optional runState,run:read,http+websocket
  listRuns,filter.status?/filter.limit?/filter.workflow?,Run summaries,run:read,http+websocket
  listWorkflows,filter.hasUi?,Workflow summaries,run:read,http+websocket
  listApprovals,filter.runId?/filter.workflow?/filter.limit?,Pending approvals,run:read,http+websocket
  streamRunEvents,runId/afterSeq?,{streamId/runId/afterSeq/currentSeq},run:read,websocket
  streamDevTools,runId/afterSeq?/fromSeq?,{streamId/runId/fromSeq/afterSeq} + devtools.event frames,observability:read,websocket
  getDevToolsSnapshot,runId/frameNo?,DevTools snapshot payload,observability:read,http+websocket
  getNodeOutput,runId/nodeId/iteration?,NodeOutputResponse,run:read,http+websocket
  getNodeDiff,runId/nodeId/iteration?,Node diff response,run:read,http+websocket
  cronList,filter.workflow?,Cron rows,cron:read,http+websocket
  cronCreate,workflow/pattern/cronId?/enabled?,Created cron row,cron:write,http+websocket
  cronDelete,cronId,{cronId/removed},cron:write,http+websocket
  cronRun,cronId? or workflow/input?,{runId/workflow},cron:write,http+websocket
  listAccounts,,Registered agent accounts (api keys redacted),account:read,http+websocket
  listMemoryFacts,namespace?,Memory facts,memory:read,http+websocket
  listScores,runId/nodeId?,Scorer results,score:read,http+websocket
  listTickets,kind?,Work docs,ticket:read,http+websocket
  createTicket,path/content/kind?/status?,Created doc row,ticket:write,http+websocket
  updateTicket,path/content?/status?,Updated doc row,ticket:write,http+websocket
  deleteTicket,path,{path/deleted},ticket:write,http+websocket
```

`health` remains available as a utility RPC and `GET /health` is available without auth. The legacy method names are still accepted for compatibility (`runs.create`, `runs.get`, `runs.list`, `runs.cancel`, `runs.rerun`, `runs.diff`, `frames.list`, `frames.get`, `attempts.list`, `attempts.get`, `workflows.list`, `approvals.list`, `approvals.decide`, `signals.send`, `cron.list`, `cron.add`, `cron.remove`, `cron.trigger`, `jumpToFrame`, `devtools.jumpToFrame`, `devtools.getNodeOutput`, `devtools.getNodeDiff`), but new clients should use the v1 names above.

### Scopes

```toon
scopes[14]{scope,allows}:
  run:read,Read run state/lists/event streams/node output/node diffs
  run:write,Launch/resume/cancel runs; implies run:read
  run:admin,Hijack or rewind runs; implies run:write and run:read
  approval:submit,Submit approval decisions
  signal:submit,Submit workflow signals
  cron:read,List cron schedules
  cron:write,Create/delete/trigger cron schedules; implies cron:read
  account:read,List registered agent accounts (api keys redacted)
  memory:read,List cross-run memory facts
  prompt:read,List workspace prompts
  score:read,List scorer/eval results for a run
  ticket:read,List work docs (tickets/plans/specs/proposals)
  ticket:write,Create/update/soft-delete work docs; implies ticket:read
  observability:read,Read DevTools and other observability streams
```

`*` grants every scope. Pass a method name string in the `scopes` array (e.g. `"launchRun"`) to grant access to exactly that RPC call. Legacy wildcard method grants such as `cron.*` continue to match legacy method names; typed scopes are the contract to use for new integrations. Legacy ranked grants (`read`, `execute`, `approve`, `admin`) are accepted so older tokens keep working.

### `rewindRun` (destructive rewind)

Rewinds a run to a prior frame and makes it resumable from that point.
This is destructive: it truncates frames, attempts, output rows, and
diff-cache entries beyond the target; reverts JJ sandboxes; marks the
run `running` again; and emits a `TimeTravelJumped` event so
`streamDevTools` subscribers rebaseline.

Caller identity is authorized per-request: the connection must have
`run:admin` scope and must also be the run owner (`userId` matches
`ownerId`) or have `role: "admin"`. Scope alone never grants access.
The legacy aliases `jumpToFrame` and `devtools.jumpToFrame` route to
`rewindRun`.

Request:

```ts
type RewindRunRequest = {
  runId: string;     // /^[a-z0-9_-]{1,64}$/
  frameNo: number;   // 0 <= frameNo <= latestFrameNo
  confirm: true;     // must be literal true
};
```

Response (`JumpResult`):

```ts
type JumpResult = {
  ok: true;
  newFrameNo: number;
  revertedSandboxes: number;
  deletedFrames: number;
  deletedAttempts: number;
  invalidatedDiffs: number;
  durationMs: number;
};
```

Also broadcast after the DB commit as `run.time_travel_jumped` with
`{ runId, fromFrameNo, toFrameNo, timestampMs, caller }`.

Quota: 10 rewinds per run per caller per hour (default window). Exceeded
→ `RateLimited`.

Failure modes and HTTP status:

| Code                   | Meaning                                                                   | HTTP  |
| ---------------------- | ------------------------------------------------------------------------- | ----- |
| `InvalidRunId`         | `runId` fails `/^[a-z0-9_-]{1,64}$/`.                                     | `400` |
| `InvalidFrameNo`       | `frameNo` is not a non-negative i32 integer.                              | `400` |
| `ConfirmationRequired` | Caller omitted `confirm: true`.                                           | `400` |
| `FrameOutOfRange`      | `frameNo` > latest frame, or run has no frames.                           | `400` |
| `Unauthorized`         | Caller is neither the run owner nor an admin (audit row still written).   | `401` |
| `RunNotFound`          | `runId` does not exist.                                                   | `404` |
| `Busy`                 | Another rewind is in flight for this run.                                 | `409` |
| `RateLimited`          | Caller exceeded rewind quota (default 10/hour).                           | `429` |
| `UnsupportedSandbox`   | A sandbox cannot be reverted (missing / untrackable `jjPointer`).         | `501` |
| `VcsError`             | A JJ revert call failed; DB/reconciler rolled back.                       | `500` |
| `RewindFailed`         | Rewind failed and rollback was partial; run marked `needs_attention`.     | `500` |

Every call, whether success, failure, or unauthorized, writes one row to
`_smithers_time_travel_audit` with `result ∈ { success, failed, partial, in_progress }`.
An in-progress row is inserted before any mutation and updated in place
on completion; startup recovery flips any leftover `in_progress` rows to
`partial`.

### Node output

`getNodeOutput` returns the DevTools Output-tab payload for a single task iteration:

```ts
type NodeOutputResponse = {
  status: "produced" | "pending" | "failed";
  row: Record<string, unknown> | null;
  schema: OutputSchemaDescriptor | null;
  partial?: Record<string, unknown> | null; // only when status === "failed"
};

type OutputSchemaDescriptor = {
  fields: Array<{
    name: string;
    type: "string" | "number" | "boolean" | "object" | "array" | "null" | "unknown";
    optional: boolean;
    nullable: boolean;
    description?: string;
    enum?: readonly unknown[];
  }>;
};
```

### Error codes

Gateway v1 RPC errors use stable code strings and HTTP status mappings:

```toon
errors[22]{code,http}:
  InvalidRequest,400
  InvalidInput,400
  Unauthorized,401
  Forbidden,403
  RunNotFound,404
  RUN_NOT_ACTIVE,409
  CronNotFound,404
  TicketNotFound,404
  NodeNotFound,404
  IterationNotFound,404
  NodeHasNoOutput,404
  FrameOutOfRange,400
  SeqOutOfRange,400
  Busy,409
  AlreadyDecided,409
  RateLimited,429
  PayloadTooLarge,413
  BackpressureDisconnect,429
  UnsupportedSandbox,501
  VcsError,500
  RewindFailed,500
  Internal,500
```

The table above is the canonical v1 registry used by the SDK and OpenAPI.
Current server responses can also surface legacy aliases from older Gateway
paths. Treat these aliases by their meaning and HTTP status; new clients should
prefer canonical codes where returned and tolerate aliases on older paths:

```toon
legacyErrors[12]{code,meaning,http}:
  INVALID_REQUEST,Invalid request,400
  INVALID_INPUT,Invalid input,400
  INVALID_FRAME,Invalid frame,400
  PROTOCOL_UNSUPPORTED,Unsupported protocol,400
  UNAUTHORIZED,Unauthorized,401
  FORBIDDEN,Forbidden,403
  NOT_FOUND,Not found,404
  METHOD_NOT_FOUND,Unknown method,404
  PAYLOAD_TOO_LARGE,Payload too large,413
  InvalidRunId,Invalid run id,400
  InvalidFrameNo,Invalid frame number,400
  ConfirmationRequired,Confirmation required,400
```

### Versioned wire shapes

All DevTools wire types carry `version: 1`.

`DevToolsSnapshot` (v1):

```ts
type DevToolsSnapshot = {
  version: 1;
  runId: string;
  frameNo: number;   // latest frame reflected in this tree
  seq: number;       // monotonic sequence id (equals frameNo today)
  root: DevToolsNode;
};

type DevToolsNode = {
  id: number;        // stable across frames for the same logical node
  type: "workflow" | "task" | "sequence" | "parallel" | /* …see protocol */;
  name: string;
  props: Record<string, unknown>;
  task?: { nodeId: string; kind: "agent" | "compute" | "static"; /* … */ };
  children: DevToolsNode[];
  depth: number;
};
```

`DevToolsDelta` (v1):

```ts
type DevToolsDelta = {
  version: 1;
  baseSeq: number;   // must match the subscriber's current seq
  seq: number;       // new seq after applying ops, in order
  ops: Array<
    | { op: "addNode"; parentId: number; index: number; node: DevToolsNode }
    | { op: "removeNode"; id: number }
    | { op: "updateProps"; id: number; props: Record<string, unknown> }
    | { op: "updateTask"; id: number; task: DevToolsNode["task"] }
    | { op: "replaceRoot"; node: DevToolsNode } // emitted when the root's
                                                // identity or shape changes;
                                                // `removeNode` of the root is
                                                // never emitted.
  >;
};
```

`DevToolsEvent` (v1), frames pushed over `devtools.event`:

```ts
type DevToolsEvent =
  | { version: 1; kind: "snapshot"; snapshot: DevToolsSnapshot }
  | { version: 1; kind: "delta"; delta: DevToolsDelta };
```

A subscription always starts with a `snapshot` event, then emits `delta` events
per frame. The server re-baselines (emits a full `snapshot` instead of a
`delta`) after 50 delta events, when a delta is larger than a fresh snapshot,
or when the gateway observes `TimeTravelJumped` for the run.

## WebSocket protocol

Three frame types share the same socket:

- `req`: `{ type: "req", id, method, params? }` from client.
- `res`: `{ type: "res", id, ok, payload?, error? }` from server, correlated by `id`.
- `event`: `{ type: "event", event, payload?, seq, stateVersion }` server-pushed; `seq` is per connection, `stateVersion` is global.

Handshake: on connect the server immediately pushes `connect.challenge` (`{ nonce, ts }`). The client replies with a `connect` request carrying `minProtocol`, `maxProtocol`, `client` metadata, `auth`, and an optional `subscribe: string[]` to filter events by `runId`. The server returns a `hello` payload (`protocol`, `features`, `policy.heartbeatMs`, `auth` with `sessionToken`/`role`/`scopes`/`userId`, `snapshot`).

After `connect`, the gateway emits `tick` events every `heartbeatMs`. `launchRun`, `submitApproval`, `submitSignal`, and `cronRun` automatically subscribe the connection to the affected `runId`. Server-pushed event names:

| Event | Category |
|---|---|
| `connect.challenge` | Connection |
| `tick` | Connection |
| `run.event` | Run lifecycle |
| `run.heartbeat` | Run lifecycle |
| `run.gap_resync` | Run lifecycle |
| `run.error` | Run lifecycle |
| `run.completed` | Run lifecycle |
| `run.time_travel_jumped` | Run lifecycle |
| `node.started` | Run lifecycle |
| `node.finished` | Run lifecycle |
| `node.failed` | Run lifecycle |
| `task.output` | Run lifecycle |
| `task.heartbeat` | Run lifecycle |
| `approval.requested` | Approval |
| `approval.decided` | Approval |
| `approval.auto_approved` | Approval |
| `cron.triggered` | Cron |
| `devtools.event` | DevTools |

For stateless callers, `POST /rpc` accepts the same body shape (`{ id, method, params }`) and returns the same `ResponseFrame`. Auth headers: `Authorization: Bearer <token>` or `x-smithers-key: <token>` (or trusted-proxy headers in trusted-proxy mode).

## GatewayOptions

```ts
type GatewayOptions = {
  protocol?: number;                 // default 1
  features?: string[];               // default ["streaming", "runs"]
  heartbeatMs?: number;              // default 15_000
  auth?: GatewayAuthConfig;
  workspaceRoot?: string;            // anchors disk-backed reads and the advertised identity
  identity?: {                       // advertised on GET /health, the health RPC, and the WS hello
    backend?: string;                // sqlite | pglite | postgres
    version?: string;                // serving package version
  };
  ui?: GatewayUiConfig;              // custom gateway UI; true mounts the built-in console
  operatorUi?: GatewayOperatorUiConfig | false; // default { path: "/console" }; false disables
  defaults?: {
    cliAgentTools?: "all" | "explicit-only";
    outOfProcessEventBridge?: boolean;
    outOfProcessEventBridgePollMs?: number;
  };
  maxBodyBytes?: number;             // default 1_048_576 for POST /rpc
  maxPayload?: number;               // default 1_048_576 for WebSocket frames
  maxConnections?: number;           // default 1_000
  eventWindowSize?: number;          // default 10_000 per-run replay frames
  outOfProcessEventBridge?: boolean; // default true; streams persisted events from detached runs
  outOfProcessEventBridgePollMs?: number; // default 1_000
  headersTimeout?: number;           // default 30_000
  requestTimeout?: number;           // default 60_000
};

type GatewayOperatorUiConfig = {
  path?: string;                      // default "/console"
  title?: string;
  props?: Record<string, unknown>;
};

type GatewayUiConfig =
  | true
  | {
      entry: string;
      path?: string;                  // gateway default "/"; workflow default "/workflows/<workflowKey>"
      title?: string;
      props?: Record<string, unknown>;
    };

type GatewayTokenGrant = {
  role: string;
  scopes: string[];
  userId?: string;
  tokenId?: string;
  issuedAtMs?: number;
  expiresAtMs?: number;
  revokedAtMs?: number;
};

type GatewayAuthConfig =
  | {
      mode: "token";
      tokens: Record<string, GatewayTokenGrant>;
      allowedOrigins?: string[];     // default [] (no Origin allowlist)
    }
  | {
      mode: "jwt";
      issuer: string;
      audience: string | string[];
      secret: string;                // HS256
      scopesClaim?: string;          // default "scope"
      roleClaim?: string;            // default "role"
      userClaim?: string;            // default "sub"
      defaultRole?: string;          // default "operator"
      defaultScopes?: string[];      // default [] when scope claim is absent
      clockSkewSeconds?: number;     // default 60; negative values clamp to 0
      allowedOrigins?: string[];     // default [] (no Origin allowlist)
    }
  | {
      mode: "trusted-proxy";
      allowedOrigins?: string[];     // default [] (no Origin allowlist)
      trustedHeaders?: string[];     // default ["x-user-id","x-user-scopes","x-user-role"]
      defaultRole?: string;          // default "operator"
      defaultScopes?: string[];      // trusted-proxy: used when the scopes header is absent, else the request is rejected
    };
```

JWT auth reads scopes from `scope`, role from `role`, and user id from `sub` unless the `*Claim` options override those claim names. Missing JWT role falls back to `defaultRole` and then `operator`; missing JWT scopes fall back to `defaultScopes` and then `[]`. Trusted-proxy auth reads `trustedHeaders` as `[user, scopes, role]`; missing role falls back to `defaultRole` and then `operator`, and missing scopes fall back to `defaultScopes`, or the request is rejected when no `defaultScopes` is configured.

`allowedOrigins` is available in every mode (token, jwt, trusted-proxy) as defense-in-depth. It defaults to `[]`, which enforces no Origin allowlist. When non-empty, the gateway rejects any HTTP RPC or WebSocket upgrade whose browser `Origin` header is not on the list; requests with no `Origin` header (server-to-server / CLI callers) are always allowed. Set it to your operator-UI origin(s) when exposing a token/jwt gateway to a browser.

Runs started through the gateway expose `ctx.auth = { triggeredBy, role, scopes, createdAt }`. `<Approval>` may further restrict decisions with `allowedScopes` and `allowedUsers`, which the gateway enforces before accepting `submitApproval`.

`headersTimeout` and `requestTimeout` are applied to the underlying Node HTTP server when `gateway.listen()` starts. Keep both below the corresponding reverse-proxy idle/read timeouts so slow clients are closed by Smithers first.

## Notes

- Identity: `GET /health`, the `health` RPC, and the WS hello carry `identity: { workspaceRoot, backend, version, pid, startedAtMs }`. Clients (the CLI's `ui`/`gateway status` among them) verify `workspaceRoot` against the workspace they resolved locally instead of trusting whichever process owns the port.
- Bind failures reject: `gateway.listen()` rejects on `EADDRINUSE` (and any other bind error) instead of crashing the process. The `smithers gateway` command retries on an ephemeral port and records the real port in its runtime state file.
- Singleton per workspace: `smithers gateway` writes `{pid, host, port, url, token, workspaceRoot, backend, version, protocol, startedAtMs}` to `<tmpdir>/smithers-gateway/<workspace-hash>.json` (mode 0600) after listening, refuses to start when a healthy gateway for the same workspace is already running, and cleans the file up on shutdown. `smithers gateway status` and `smithers gateway stop` manage it; stale files (dead pid, identity mismatch) are cleaned up on the next discovery.
- Monitor mount: the `smithers gateway` CLI singleton also serves the [Smithers Monitor](/guides/monitor), a live web UI over every run in the workspace, at `/monitor` (open it with `bunx smithers-orchestrator monitor`). The mount ships inside the CLI, so it needs no `.smithers/` pack and no registered workflow UI. Library embedders control the other UI mounts through `ui` (custom gateway UI, workflow UIs at `/workflows/<key>`) and `operatorUi` (the embeddable operator console, default `/console`).
- `--mint-token`: mints a random bearer, requires it on every request, and records it only in the runtime state file (plus the daemon's own stderr). `bunx smithers-orchestrator ui` reads the token from the state file automatically for CLI RPC calls, but browser navigation to the printed workflow UI URL cannot send that bearer header until UI token injection ships; use `smithers gateway` without `--mint-token` when you need browser-served workflow UIs. Other clients can read `SMITHERS_TOKEN` / `SMITHERS_API_KEY`.
- Cron: `gateway.register(name, wf, { schedule })` writes a cron row keyed `gateway:<name>`; the gateway polls between 1 s and 15 s (clamped from `heartbeatMs`). Cron-fired runs get `ctx.auth.role = "system"`, `triggeredBy = "cron:gateway"`, `scopes = ["*"]`.
- Host defense: an unauthenticated gateway (the autostart default) rejects any request whose `Host` header is non-loopback, as a DNS-rebinding defense, so binding `--host 0.0.0.0` without a token returns 403 `Host is not allowed`. This applies only when no `auth` is configured. To deliberately expose an unauthenticated remote bind, pass `smithers gateway --insecure` (or `up --serve --insecure`), which trusts any Host; `SMITHERS_GATEWAY_TRUST_ANY_HOST=1` (gateway) and `SMITHERS_SERVE_TRUST_ANY_HOST=1` (serve) do the same via the environment.
- JWT mode currently validates `alg=HS256`, HMAC, `iss`, `aud`, `exp`, `nbf`. Scope claims may be arrays or space/comma-separated strings.
- Trusted-proxy mode is only safe behind something you control (Cloudflare Access, internal API gateway) that strips and rewrites identity headers.
- DevTools streams: see Versioned wire shapes for re-baseline triggers; over-capacity subscribers receive `BackpressureDisconnect`.

---

## MCP Server

> Expose Smithers as a Model Context Protocol stdio server so any MCP client (Claude Code, Cursor, Codex, or your own agent) can list, run, inspect, and control workflows without shell scripting.

Smithers ships a built-in MCP stdio server. Passing `--mcp` to the CLI speaks the Model Context Protocol over stdin/stdout instead of acting as an interactive CLI. Any MCP-aware client can connect, discover workflows, start runs, watch progress, resolve approvals, and revert bad attempts through structured tool calls.

Use the MCP server when an AI agent should drive Smithers autonomously. Use the [HTTP Server](/integrations/server) for REST endpoints for human-written code or webhooks.

---

## Setup

### Start the server

```bash
bunx smithers-orchestrator --mcp
```

This starts the semantic surface: a stable, structured tool set for AI agent consumption, documented on this page.

Two additional surfaces are available via `--surface`:

```bash
# Semantic tools only (default)
bunx smithers-orchestrator --mcp --surface semantic

# Raw CLI-mirroring tools only
bunx smithers-orchestrator --mcp --surface raw

# Both surfaces registered on the same server
bunx smithers-orchestrator --mcp --surface both
```

Use `--surface raw` only for direct CLI parity. Prefer the semantic surface for new integrations: every tool returns a `{ ok, data, error }` envelope with Zod-validated input and output schemas.

Scope the semantic surface when a client should not receive every Smithers control tool:

```bash
# Expose only selected semantic tools
bunx smithers-orchestrator --mcp --allowed-tools list_workflows,get_run

# Expose only tools annotated as read-only
bunx smithers-orchestrator --mcp --read-only
```

`--allowed-tools` accepts a comma-separated list of semantic tool names. Passing an empty allowlist intentionally exposes no semantic tools. `--read-only` removes semantic tools with write or control side effects, such as starting runs, resolving approvals, or reverting attempts. With `--surface both`, these controls apply to the semantic toolset only; raw CLI-mirroring tools are still registered by the raw surface.

### Register manually

For clients that read JSON config directly:

```json
{
  "mcpServers": {
    "smithers": {
      "command": "bunx",
      "args": ["smithers-orchestrator", "--mcp"]
    }
  }
}
```

Project-scoped install (e.g. a monorepo where Smithers is a dev dependency; ensure `smithers-orchestrator` is in the local `package.json`):

```json
{
  "mcpServers": {
    "smithers": {
      "command": "bunx",
      "args": ["smithers-orchestrator", "--mcp"]
    }
  }
}
```

### If `mcp add` fails

`bunx smithers-orchestrator mcp add` hands the launch command to a registration helper that expects it as a single argument. If a runner or shell word-splits it, the helper sees a bare `--mcp` token and aborts:

```
Registering MCP server...
code: MCP_ADD_FAILED
message: "error: unknown option '--mcp'"
```

Register with your agent's own CLI instead. The `--` separator tells the agent that everything after it is the launch command, so it never parses `--mcp` as one of its own flags:

```bash
codex mcp add smithers -- bunx smithers-orchestrator --mcp
claude mcp add smithers -- bunx smithers-orchestrator --mcp
```

Any MCP-aware CLI follows the same `<agent> mcp add <name> -- <command>` shape. Or write the JSON or TOML config by hand using the snippets above. The Smithers CLI prints these fallback commands automatically whenever `mcp add` fails.

---

## Tool Registration

On start, each tool is registered with its input schema, output schema, and MCP annotations. Every tool carries:

- **`inputSchema`**: Zod object describing accepted parameters.
- **`outputSchema`**: Zod schema for the structured response envelope.
- **`annotations`**: MCP annotation metadata (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`).

### Structured tool envelope

Every tool returns the same shape:

```ts
{
  ok: boolean;
  data?: { ... };     // present on success
  error?: {           // present on failure
    code: string;
    message: string;
    details?: Record<string, unknown> | null;
    docsUrl?: string | null;
  };
}
```

The response is also echoed as a `text` content block, so clients that do not parse `structuredContent` still receive the JSON payload.

### Tool annotations

| Annotation | Tools | Meaning |
|---|---|---|
| `readOnlyHint: true` | Most query tools | Tool does not modify state |
| `readOnlyHint: false, openWorldHint: true` | `run_workflow` | Launches external processes |
| `readOnlyHint: false, destructiveHint: true, idempotentHint: false` | `resolve_approval`, `revert_attempt`, `rewind_run`, `restore_checkpoint`, `time_travel` | Mutates persisted state irreversibly |
| `readOnlyHint: false, idempotentHint: false` | `fork_run`, `replay_run` | Creates new run/branch state |

---

## Tool Reference

### list_workflows

List all Smithers workflows discovered in the working directory.

**Input:** none

**Output:**

```ts
{
  workflows: Array<{
    id: string;
    metadataVersion: number;
    displayName: string;
    scope: "local" | "global";
    entryFile: string;
    path: string;
    sourceType: string;
    description: string;
    tags: string[];
    aliases: string[];
  }>;
}
```

Use the returned `id` values as the `workflowId` parameter for `run_workflow`.

---

### run_workflow

Start or resume a discovered workflow.

**Input:**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `workflowId` | `string` | required | Workflow ID from `list_workflows` |
| `input` | `Record<string, unknown>` | `{}` | Workflow input object |
| `prompt` | `string` | - | Shorthand: sets `input.prompt` when `input` is not provided |
| `runId` | `string` | auto | Custom run ID |
| `resume` | `boolean` | `false` | Resume an existing run; requires `runId` |
| `force` | `boolean` | `false` | Force-start even if a run with this ID already exists |
| `waitForTerminal` | `boolean` | `false` | Block until the run reaches a terminal state |
| `waitForStartMs` | `number` | `1000` | For background launches, how long to wait for the run row to appear in the database |
| `maxConcurrency` | `number` | - | Max concurrent nodes |
| `rootDir` | `string` | - | Root directory for tool sandboxing and path resolution |
| `logDir` | `string` | - | Directory for log files |
| `allowNetwork` | `boolean` | `false` | Allow network access in `bash` tool |
| `maxOutputBytes` | `number` | - | Cap on node output size |
| `toolTimeoutMs` | `number` | - | Per-tool call timeout |
| `hot` | `boolean` | `false` | Enable hot-reloading of the workflow file |

**Output:**

```ts
{
  workflow: {
    id: string;
    metadataVersion: number;
    displayName: string;
    scope: "local" | "global";
    entryFile: string;
    path: string;
    sourceType: string;
    description: string;
    tags: string[];
    aliases: string[];
  };
  runId: string;
  launchMode: "background" | "waited";
  requestedResume: boolean;
  status: string;
  observedRun: RunSummary | null;
  result: { runId, status, output?, error? } | null;
}
```

**Background vs. waited launch**

By default (`waitForTerminal: false`) the tool fires the workflow and returns immediately with `launchMode: "background"`. `observedRun` reflects the run state polled during `waitForStartMs`. Use `watch_run` to track progress.

Set `waitForTerminal: true` to block until the workflow finishes. `result` is populated and `launchMode` is `"waited"`.

**Run option forwarding**

`rootDir`, `logDir`, `allowNetwork`, `maxOutputBytes`, `toolTimeoutMs`, and `hot` are forwarded verbatim to `runWorkflow`. They override values baked into the workflow file.

---

### list_runs

List recent runs with summary data.

**Input:**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `limit` | `number` (1–200) | `20` | Max runs to return |
| `status` | `string` | - | Filter by status (`running`, `finished`, `failed`, etc.) |

**Output:**

```ts
{
  runs: RunSummary[];
}
```

`RunSummary` fields: `runId`, `workflowName`, `workflowPath`, `parentRunId`, `status`, `createdAtMs`, `startedAtMs`, `finishedAtMs`, `heartbeatAtMs`, `activeNodeId`, `activeNodeLabel`, `pendingApprovalCount`, `waitingTimers`, `countsByState`.

---

### get_run

Get the full detail record for a specific run, including steps, approvals, timers, loop state, lineage, config, and error.

**Input:**

| Parameter | Type | Description |
|---|---|---|
| `runId` | `string` | Run ID |

**Output:**

```ts
{
  run: RunSummary & {
    steps: Array<{ nodeId, iteration, state, lastAttempt, updatedAtMs, outputTable, label }>;
    approvals: PendingApproval[];
    loops: Array<{ loopId, iteration, maxIterations }>;
    continuedFromRunIds: string[];
    activeDescendantRunId: string | null;
    config: unknown | null;
    error: unknown | null;
  };
}
```

---

### watch_run

Poll a run at a fixed interval until it reaches a terminal state or a timeout expires.

**Input:**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `runId` | `string` | required | Run to watch |
| `intervalMs` | `number` | `1000` | Poll interval (minimum enforced by runtime) |
| `timeoutMs` | `number` | `30000` | Wall-clock budget before giving up |

**Output:**

```ts
{
  runId: string;
  intervalMs: number;
  pollCount: number;
  reachedTerminal: boolean;
  timedOut: boolean;
  finalRun: RunSummary;
  snapshots: Array<{ observedAtMs: number; run: RunSummary }>;
}
```

When `timedOut` is `true` the run is still active, so call `watch_run` again or raise `timeoutMs`. Terminal statuses: any status other than `running`, `waiting-approval`, `waiting-event`, or `waiting-timer`, including `finished`, `failed`, `cancelled`, and `continued`.

---

### explain_run

Return a structured diagnosis explaining why a run is blocked, waiting, or stale.

**Input:**

| Parameter | Type | Description |
|---|---|---|
| `runId` | `string` | Run ID |

**Output:**

```ts
{
  diagnosis: {
    runId: string;
    status: string;
    summary: string;
    generatedAtMs: number;
    blockers: Array<{
      kind: string;
      nodeId: string;
      iteration: number | null;
      reason: string;
      waitingSince: number;
      unblocker: string;
      context?: string;
      signalName?: string | null;
      dependencyNodeId?: string | null;
      firesAtMs?: number | null;
      remainingMs?: number | null;
      attempt?: number | null;
      maxAttempts?: number | null;
    }>;
    currentNodeId: string | null;
  };
}
```

`summary` is a human-readable sentence. `blockers` lists every node preventing progress; `unblocker` describes what action or event would unblock it.

---

### list_pending_approvals

List approvals that are waiting for a human decision, optionally filtered by run, workflow, or node.

**Input:** All parameters optional. Omit all to list every pending approval across all runs.

| Parameter | Type | Description |
|---|---|---|
| `runId` | `string` | Filter by run ID |
| `workflowName` | `string` | Filter by workflow name |
| `nodeId` | `string` | Filter by node ID |

**Output:**

```ts
{
  approvals: Array<{
    runId: string;
    nodeId: string;
    iteration: number;
    status: string;
    requestedAtMs: number | null;
    decidedAtMs: number | null;
    note: string | null;
    decidedBy: string | null;
    request: unknown;
    decision: unknown;
    autoApproved?: boolean;
    workflowName: string | null;
    runStatus: string | null;
    nodeLabel: string | null;
  }>;
}
```

---

### resolve_approval

Approve or deny a pending approval. This tool is destructive and non-idempotent.

**Input:**

| Parameter | Type | Description |
|---|---|---|
| `action` | `"approve" \| "deny"` | required, decision to record |
| `runId` | `string` | Filter to a specific run |
| `workflowName` | `string` | Filter by workflow name |
| `nodeId` | `string` | Filter by node ID |
| `iteration` | `number` | Filter by loop iteration |
| `note` | `string` | Optional note to record with the decision |
| `decidedBy` | `string` | Identity of the decision-maker |
| `decision` | `unknown` | Structured decision payload passed back to the workflow |

**Ambiguity guard**

Zero matches errors with `INVALID_INPUT`. More than one match errors with `INVALID_INPUT` and returns matches in `details.matches`; add `runId`, `nodeId`, or `iteration` to narrow the selection. The tool never guesses when multiple approvals match.

**Output:**

```ts
{
  action: "approve" | "deny";
  approval: PendingApproval;   // with updated status, decidedAtMs, note, decidedBy
  run: RunSummary | null;
}
```

---

### ask_human

Block the current run and ask a human to make a decision, then wait for their answer. Use this whenever the agent is blocked, uncertain, missing information, or about to take an irreversible or destructive action, instead of guessing. The tool creates a durable, pending human request and returns only once it is resolved.

When run inside a Smithers task, the run/node context is taken from the `SMITHERS_RUN_ID` / `SMITHERS_NODE_ID` / `SMITHERS_ITERATION` environment variables Smithers injects into the agent; pass `runId`/`nodeId`/`iteration` explicitly to override, or rely on single-active-run autodetection.

The orchestrating agent resolves the request on the human's behalf: relay the question to the human in conversation, collect their decision, then run `bunx smithers-orchestrator human answer <requestId> --value '<json>'` (or `bunx smithers-orchestrator human cancel <requestId>`) yourself; never instruct the human to run these. `bunx smithers-orchestrator human inbox` lists everything waiting.

**Input:**

| Parameter | Type | Description |
|---|---|---|
| `prompt` | `string` | required, the decision or question to put to a human |
| `context` | `string` | Extra context appended to the prompt |
| `choices` | `string[]` | Fixed choices; restricts the human's answer to one of these |
| `runId` | `string` | Run to attach to (default: `SMITHERS_RUN_ID` or the single active run) |
| `nodeId` | `string` | Node to attach to (default: `SMITHERS_NODE_ID`) |
| `iteration` | `number` | Loop iteration (default: `SMITHERS_ITERATION` or 0) |
| `timeoutSeconds` | `number` | Seconds before the request expires (0/unset = no timeout) |
| `pollSeconds` | `number` | Poll interval while blocking (default 3s) |

**Output:**

```ts
{
  requestId: string;
  runId: string;
  nodeId: string;
  iteration: number;
  status: "answered" | "cancelled" | "expired" | "missing" | "aborted";
  decision: "approved" | "blocked";   // "blocked" => do not proceed
  response: unknown | null;            // the human's answer when status is "answered"
  answeredBy: string | null;
}
```

---

### get_node_detail

Get enriched detail for a single node, including all attempts, tool calls, token usage, scorer results, and validated output.

**Input:**

| Parameter | Type | Description |
|---|---|---|
| `runId` | `string` | required |
| `nodeId` | `string` | required |
| `iteration` | `number` | Loop iteration (default: latest) |

**Output:**

```ts
{
  detail: {
    node: { runId, nodeId, iteration, state, lastAttempt, updatedAtMs, outputTable, label };
    status: string;
    durationMs: number | null;
    attemptsSummary: { total, failed, cancelled, succeeded, waiting };
    attempts: unknown[];
    toolCalls: unknown[];
    tokenUsage: unknown;
    scorers: unknown[];
    output: {
      validated: unknown | null;
      raw: unknown | null;
      source: "cache" | "output-table" | "none";
      cacheKey: string | null;
    };
    approval: PendingApproval | null;
    limits: {
      toolPayloadBytesHuman: number;
      validatedOutputBytesHuman: number;
    };
  };
}
```

---

### revert_attempt

Revert the workspace and frame history back to the state captured at a specific attempt. This is destructive and non-idempotent.

**Input:**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `runId` | `string` | required | Run containing the node |
| `nodeId` | `string` | required | Node to revert |
| `iteration` | `number` | `0` | Loop iteration |
| `attempt` | `number` | required | Attempt number to revert to (must be ≥ 1) |

**Output:**

```ts
{
  runId: string;
  nodeId: string;
  iteration: number;
  attempt: number;
  success: boolean;
  error?: string;
  jjPointer?: string;
  run: RunSummary | null;
}
```

---

### fork_run

Create a branched run from a time-travel snapshot checkpoint without starting it.

**Input:**

| Parameter | Type | Description |
|---|---|---|
| `parentRunId` | `string` | Source run ID |
| `frameNo` | `number` | Snapshot frame number |
| `resetNodes` | `string[]` | Node IDs to reset to pending in the fork |
| `inputOverrides` | `Record<string, unknown>` | Input fields to overlay on the snapshot input |
| `branchLabel` | `string` | Optional branch label |

**Output:** `{ runId, parentRunId, parentFrameNo, branch, snapshot, run }`

---

### replay_run

Fork a run from a checkpoint for replay, optionally restoring VCS state. Resume the returned `runId` with `run_workflow` when needed.

**Input:** same as `fork_run`, plus:

| Parameter | Type | Default | Description |
|---|---|---|---|
| `restoreVcs` | `boolean` | `false` | Restore the working copy to the source frame revision |
| `cwd` | `string` | - | Working directory used for VCS restore |

**Output:** `{ runId, parentRunId, parentFrameNo, branch, snapshot, vcsRestored, vcsPointer, vcsError?, run }`

---

### rewind_run

Rewind a run to a previous frame, deleting later frames and invalidating derived state. This is destructive and requires `confirm: true`.

**Input:**

| Parameter | Type | Description |
|---|---|---|
| `runId` | `string` | Run to rewind |
| `frameNo` | `number` | Target frame number |
| `confirm` | `boolean` | Must be `true` |

**Output:** `{ result, run }`

---

### restore_checkpoint

Restore the worktree to a durability checkpoint for a node. If `seq` is omitted, the latest matching checkpoint is used.

**Input:**

| Parameter | Type | Description |
|---|---|---|
| `runId` | `string` | Run containing the checkpoint |
| `nodeId` | `string` | Node whose checkpoint should be restored |
| `iteration` | `number` | Optional loop iteration |
| `seq` | `number` | Optional checkpoint sequence |

**Output:** `{ runId, nodeId, iteration, seq, commitId, cwd, success, error? }`

---

### list_snapshots

List durability workspace checkpoints for a run with matching VCS operation IDs when available.

**Input:** `{ runId: string }`

**Output:** `{ snapshots: Array<{ seq, nodeId, iteration, attempt, tier, source, label, commitId, operationId, cwd, createdAtMs }> }`

---

### get_timeline

Return the time-travel timeline for a run, optionally including all child forks recursively.

**Input:**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `runId` | `string` | required | Run ID |
| `tree` | `boolean` | `false` | Include child forks recursively |

**Output:** `{ timeline: unknown }`

---

### time_travel

Reset a run back to a prior node attempt and optionally restore VCS state. If the run is still marked running, pass `force: true`.

**Input:**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `runId` | `string` | required | Run ID |
| `nodeId` | `string` | required | Node to travel back to |
| `iteration` | `number` | `0` | Loop iteration |
| `attempt` | `number` | latest | Attempt number |
| `restoreVcs` | `boolean` | `true` | Restore filesystem state |
| `resetDependents` | `boolean` | `true` | Reset dependent nodes too |
| `force` | `boolean` | `false` | Allow time travel when the run is still running |

**Output:** `{ result, run }`

---

### list_artifacts

List structured output artifacts produced by nodes in a run.

**Input:**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `runId` | `string` | required | Run ID |
| `nodeId` | `string` | - | Limit to a specific node |
| `includeRaw` | `boolean` | `false` | Include raw (pre-validation) output values |

**Output:**

```ts
{
  artifacts: Array<{
    artifactId: string;   // "<runId>:<nodeId>:<iteration>"
    kind: "node-output";
    runId: string;
    nodeId: string;
    iteration: number;
    label: string | null;
    state: string;
    outputTable: string | null;
    source: "cache" | "output-table" | "none";
    cacheKey: string | null;
    value: unknown | null;
    rawValue?: unknown | null;   // only when includeRaw=true
  }>;
}
```

Only nodes with an `outputTable` and a non-`none` output source are included.

---

### get_chat_transcript

Return the structured agent chat transcript for a run, grouped by attempts.

**Input:**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `runId` | `string` | required | Run ID |
| `all` | `boolean` | `false` | Include all attempts, not just those with known output events |
| `includeStderr` | `boolean` | `true` | Include stderr messages |
| `tail` | `number` | - | Return only the last N messages |

**Output:**

```ts
{
  runId: string;
  attempts: Array<{
    attemptKey: string;
    nodeId: string;
    iteration: number;
    attempt: number;
    state: string;
    startedAtMs: number;
    finishedAtMs: number | null;
    cached: boolean;
    meta: unknown | null;
  }>;
  messages: Array<{
    id: string;
    attemptKey: string;
    nodeId: string;
    iteration: number;
    attempt: number;
    role: "user" | "assistant" | "stderr";
    stream: "stdout" | "stderr" | null;
    timestampMs: number;
    text: string;
    source: "prompt" | "event" | "responseText";
  }>;
}
```

Messages are sorted by `timestampMs`. Use `tail` to limit context window usage on long transcripts.

---

### get_run_events

Return the raw structured event history for a run with optional filtering.

**Input:**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `runId` | `string` | required | Run ID |
| `afterSeq` | `number` | - | Only events with `seq` greater than this value |
| `limit` | `number` (1–10000) | `200` | Max events to return |
| `nodeId` | `string` | - | Filter to events for a specific node |
| `types` | `string[]` | - | Filter to specific event types (e.g. `["NodeFinished", "NodeFailed"]`) |
| `sinceTimestampMs` | `number` | - | Only events at or after this timestamp |

**Output:**

```ts
{
  runId: string;
  events: Array<{
    runId: string;
    seq: number;
    timestampMs: number;
    type: string;
    payload: unknown | null;
  }>;
}
```

Paginate via `afterSeq`: pass the `seq` of the last received event to fetch the next page.

---

## Usage Examples

### List workflows and start a run

```
> list_workflows {}

{
  "ok": true,
  "data": {
    "workflows": [
      { "id": "bugfix", "displayName": "bugfix", "entryFile": "./workflows/bugfix.tsx", "sourceType": "user" }
    ]
  }
}

> run_workflow { "workflowId": "bugfix", "prompt": "Fix the auth token expiry bug" }

{
  "ok": true,
  "data": {
    "runId": "smi_abc123",
    "launchMode": "background",
    "status": "running",
    ...
  }
}
```

### Watch until complete

```
> watch_run { "runId": "smi_abc123", "timeoutMs": 120000 }

{
  "ok": true,
  "data": {
    "reachedTerminal": true,
    "timedOut": false,
    "finalRun": { "status": "finished", ... }
  }
}
```

### Resolve a pending approval

```
> list_pending_approvals { "runId": "smi_abc123" }

{
  "ok": true,
  "data": {
    "approvals": [
      { "nodeId": "deploy", "iteration": 0, "nodeLabel": "Deploy to production", ... }
    ]
  }
}

> resolve_approval { "action": "approve", "runId": "smi_abc123", "nodeId": "deploy", "decidedBy": "alice", "note": "Looks good" }

{
  "ok": true,
  "data": {
    "action": "approve",
    "approval": { "status": "approved", "decidedAtMs": 1707500100000, ... },
    "run": { "status": "running", ... }
  }
}
```

### Debug a blocked run

```
> explain_run { "runId": "smi_abc123" }

{
  "ok": true,
  "data": {
    "diagnosis": {
      "summary": "Run is waiting for a human approval on node 'deploy'.",
      "blockers": [
        {
          "kind": "approval",
          "nodeId": "deploy",
          "reason": "Node requires human approval before proceeding.",
          "unblocker": "Call resolve_approval with action=approve or action=deny."
        }
      ]
    }
  }
}
```

### Revert a failed attempt

```
> get_node_detail { "runId": "smi_abc123", "nodeId": "analyze" }

{
  "ok": true,
  "data": {
    "detail": {
      "attemptsSummary": { "total": 3, "failed": 2, "succeeded": 1 },
      ...
    }
  }
}

> revert_attempt { "runId": "smi_abc123", "nodeId": "analyze", "attempt": 1 }

{
  "ok": true,
  "data": {
    "success": true,
    "run": { "status": "running", ... }
  }
}
```

---

## Error Codes

Errors follow the structured envelope. Common codes:

| Code | Meaning |
|---|---|
| `RUN_NOT_FOUND` | No run or workflow exists with the given ID |
| `INVALID_INPUT` | Missing required field, failed validation, or ambiguous approval filter |
| `WORKFLOW_MISSING_DEFAULT` | Workflow file has no default export |
