# Hermes Phase 5 — Dispatch Hardening · Main Implementation Plan

**Spec:** `docs/changes/hermes-phase-5-dispatch-hardening/proposal.md`
**Date:** 2026-05-16
**Estimated effort:** 3–4 weeks, sequenced across 4 task tracks

## Task Order Rationale

Tasks are ordered so each track's _types_ land before its _implementation_, and so the router extension (Track A) lands before backends that depend on the new `isolation` field. Tracks B and C are parallelizable after Track A, and Track D depends only on Track A's type additions for `costCeiling`.

```
Track A (Types + Router)   ─┬→ Track B (SSH backend)
                            ├→ Track C (Serverless backend)
                            └→ Track D (Cost ceiling)
```

Each task ends in a green `pnpm typecheck && pnpm test` before the next starts.

---

## Track A — Types + Router + Factory

Foundation. Lands `IsolationTier`, the discriminated union members, the router extension, and the factory wiring. After this track, configs _can_ declare SSH / serverless backends and `isolation` routing, but instantiation throws "not implemented" until Track B / C lands.

### Task A1 — Add `IsolationTier` + `BackendDef.isolation` field

**Files:** `packages/types/src/orchestrator.ts`, `packages/types/src/index.ts`

**Changes:**

- Add `export type IsolationTier = 'none' | 'container' | 'remote-sandbox'` to `orchestrator.ts`
- Add optional `isolation?: IsolationTier` on every existing `BackendDef` variant
- Re-export `IsolationTier` from `index.ts`

**Verification:**

- `pnpm --filter @harness-engineering/types build` succeeds
- `pnpm typecheck` from repo root succeeds

### Task A2 — Add `SshBackendDef` + `ServerlessBackendDef` union members

**Files:** `packages/types/src/orchestrator.ts`, `packages/types/src/index.ts`

**Changes:**

- Add `SshBackendDef { type: 'ssh'; host: string; user?: string; port?: number; identityFile?: string; remoteCommand: string; isolation?: IsolationTier; sshOptions?: string[] }`
- Add `ServerlessBackendDef { type: 'serverless'; adapter: 'oci'; image: string; registry?: string; pullPolicy?: 'always' | 'if-not-present' | 'never'; isolation?: IsolationTier; envPassthrough?: string[] }`
- Extend `BackendDef = ... | SshBackendDef | ServerlessBackendDef`
- Re-export both from `index.ts`

**Verification:**

- `pnpm typecheck` succeeds (exhaustive switch in `createBackend` will go red — fixed in next task)

### Task A3 — Extend `RoutingUseCase` + `RoutingConfig` with isolation axis

**Files:** `packages/types/src/orchestrator.ts`

**Changes:**

- Add `{ kind: 'isolation'; tier: IsolationTier }` member to `RoutingUseCase`
- Add `isolation?: { none?: string; container?: string; 'remote-sandbox'?: string }` field to `RoutingConfig`

**Verification:**

- `pnpm typecheck` succeeds

### Task A4 — Extend `BackendRouter` to resolve isolation

**Files:** `packages/orchestrator/src/agent/backend-router.ts`

**Changes:**

- Handle `case 'isolation':` in `resolve()` — read `routing.isolation?.[useCase.tier]`, fall back to `routing.default`
- Walk `routing.isolation.{none,container,remote-sandbox}` in `validateReferences()`

**Verification:**

- New router test cases pass (see Task A6)

### Task A5 — Extend `createBackend()` factory for `ssh` + `serverless`

**Files:** `packages/orchestrator/src/agent/backend-factory.ts`

**Changes:**

- Add `case 'ssh':` returning a placeholder that throws `"SshBackend not implemented yet"` (real impl in Track B)
- Add `case 'serverless':` returning a placeholder that throws `"ServerlessBackend not implemented yet"` (real impl in Track C)
- Keep `default:` exhaustive

**Verification:**

- `pnpm typecheck` succeeds; existing factory tests pass

### Task A6 — Tests for router + factory

**Files:** `packages/orchestrator/test/agent/backend-router.test.ts` (extend), `packages/orchestrator/test/agent/backend-factory.test.ts` (extend)

**New cases:**

- `resolve({ kind: 'isolation', tier: 'remote-sandbox' })` returns `routing.isolation['remote-sandbox']` when set
- Falls back to `routing.default` when not set
- `validateReferences` throws on unknown name in `routing.isolation.*`
- Factory creates placeholder for `ssh` / `serverless` types; placeholder throws on `startSession()` only

**Verification:**

- `pnpm --filter @harness-engineering/orchestrator test` green

---

## Track B — SSH Backend

Implements `SshBackend`. Depends on Track A.

### Task B1 — `SshBackend` skeleton

**Files:** `packages/orchestrator/src/agent/backends/ssh.ts` (new)

**Changes:**

- `SshBackend implements AgentBackend`
- Constructor takes a typed `SshBackendConfig` shape (mirrors `SshBackendDef` minus the discriminant)
- `name = 'ssh'`
- Implement `startSession`, `runTurn`, `stopSession`, `healthCheck` against a `child_process.spawn('ssh', […])` transport

**Implementation notes:**

- Use `node:child_process` `spawn` — no new deps
- SSH command shape: `ssh [-i identityFile] [-p port] [user@]host -- <remoteCommand> [args]`
- `runTurn` parses the same JSON-lines protocol the local backend speaks
- Process tracking via a `Map<sessionId, ChildProcess>` like other backends

**Verification:**

- `pnpm typecheck` succeeds; unit test (next task) exercises the spawn args contract via a mockable `spawnImpl` injected option

### Task B2 — `SshBackend` unit tests with injected spawn

**Files:** `packages/orchestrator/test/agent/backends/ssh.test.ts` (new)

**Cases:**

- `startSession` formats the SSH command correctly (user, port, identity, host, remoteCommand)
- `runTurn` yields events from a synthetic stream
- `stopSession` sends SIGTERM then SIGKILL on timeout
- `healthCheck` returns `Ok` when stub spawn succeeds; `Err` (`agent_not_found`) when it fails
- Rejects `host` containing `;`, `&`, `|`, backticks (command-injection guard)

**Verification:**

- `pnpm --filter @harness-engineering/orchestrator test` green

### Task B3 — Wire `SshBackend` into `createBackend()`

**Files:** `packages/orchestrator/src/agent/backend-factory.ts`

**Changes:**

- Replace the placeholder for `case 'ssh':` with real `new SshBackend(...)` construction

**Verification:**

- Factory tests pass; integration test (Track B4) green

### Task B4 — Operator docs for SSH backend

**Files:** `docs/knowledge/orchestrator/backends-ssh.md` (new)

**Contents:**

- Example `harness.config.json` block declaring an SSH backend
- Required SSH setup on the remote (agent CLI installed, model API keys configured locally)
- Key management guidance (use `~/.ssh/config`, no key in config)
- Security guardrails: `host` is shell-escaped; `--privileged`-equivalent SSH flags are blocked by config validation
- Troubleshooting: connection errors, auth failures, slow first-spawn

---

## Track C — Serverless Backend + OCI Adapter

Implements `ServerlessBackend` abstract + `OciServerlessBackend`. Parallelizable with Track B after Track A.

### Task C1 — `ServerlessBackend` abstract

**Files:** `packages/orchestrator/src/agent/backends/serverless.ts` (new)

**Changes:**

- `abstract class ServerlessBackend implements AgentBackend`
- Abstract methods: `coldStart(params): Promise<Result<ServerlessHandle, AgentError>>`, `runOnHandle(handle, turn): AsyncGenerator<AgentEvent, TurnResult>`, `teardown(handle): Promise<Result<void, AgentError>>`
- Implements `AgentBackend.startSession / runTurn / stopSession / healthCheck` in terms of those three primitives + a `Map<sessionId, ServerlessHandle>`
- `name` is abstract (subclasses set it)

**Verification:**

- `pnpm typecheck` succeeds

### Task C2 — `OciServerlessBackend` concrete adapter

**Files:** `packages/orchestrator/src/agent/backends/serverless.ts` (extend) or `serverless-oci.ts` (new)

**Changes:**

- `OciServerlessBackend extends ServerlessBackend`
- `coldStart` invokes `docker run` (or `podman run`, configurable) with `--rm -d` and returns the container id as a `ServerlessHandle`
- `runOnHandle` execs into the running container and streams events
- `teardown` sends `docker stop` + `docker rm`
- Same `BLOCKED_FLAGS` sanitization as `ContainerBackend` for `extraArgs`

**Verification:**

- Unit test with a mocked process runner

### Task C3 — `OciServerlessBackend` unit tests

**Files:** `packages/orchestrator/test/agent/backends/serverless-oci.test.ts` (new)

**Cases:**

- `coldStart` formats `docker run` args correctly (image, env passthrough, network)
- `runOnHandle` streams events from a stub
- `teardown` calls `docker stop && docker rm`
- Rejects `image` containing `;`, `&`, `|` (injection guard)
- `pullPolicy: 'never'` does _not_ invoke `docker pull`

**Verification:**

- `pnpm --filter @harness-engineering/orchestrator test` green

### Task C4 — Wire serverless into `createBackend()`

**Files:** `packages/orchestrator/src/agent/backend-factory.ts`

**Changes:**

- Replace placeholder for `case 'serverless':` with real construction
- Switch on `def.adapter` (today: only `'oci'`; future: `'modal' | 'daytona'`)

**Verification:**

- Factory tests pass

### Task C5 — Operator docs for serverless backends

**Files:** `docs/knowledge/orchestrator/backends-serverless.md` (new)

**Contents:**

- Example config for OCI adapter
- Cold-start latency expectations and when to use serverless vs. local vs. container
- Adapter authoring guide (sub-class `ServerlessBackend`, register in factory)

---

## Track D — Cost Ceiling End-to-End

Per-task cost ceiling with abort-on-exceed. Parallelizable with Tracks B and C after Track A's type additions land.

### Task D1 — Add `costCeiling` to `TaskDefinition`, `costUsd` to `RunResult`

**Files:** `packages/orchestrator/src/maintenance/types.ts`

**Changes:**

- Add `costCeiling?: { maxUsd: number; warnAtPct?: number }` to `TaskDefinition`
- Add `costUsd: number` (required, defaults to 0 at construction sites) to `RunResult`

**Verification:**

- `pnpm typecheck` succeeds; existing maintenance tests pass after the construction-site updates

### Task D2 — `pricing.ts` wrapper (orchestrator-side)

**Files:** `packages/orchestrator/src/cost/pricing.ts` (new)

**Changes:**

- Re-export the relevant `ModelPricing` shape and `computeCostUsd(usage, pricing): number` helper from `@harness-engineering/core`
- This file is the orchestrator-layer's stable surface against the core/usage internals (so backend code never imports `core/usage` directly)

**Verification:**

- `pnpm typecheck` succeeds

### Task D3 — `CostCeilingMonitor` service

**Files:** `packages/orchestrator/src/cost/cost-ceiling-monitor.ts` (new)

**Changes:**

- `class CostCeilingMonitor`
- `registerTask(taskId, ceiling)`: starts tracking
- `recordTurn(taskId, usage, modelName)`: applies pricing, accumulates, fires abort signal when exceeded
- `unregisterTask(taskId)`: stops tracking, returns final `costUsd`
- Emits `EventEmitter` events: `'abort'` (with `{ taskId, costUsd }`), `'warn'` (with `{ taskId, costUsd, pctOfCeiling }`)

**Verification:**

- Unit tests (next task) exercise the state machine

### Task D4 — `CostCeilingMonitor` unit tests

**Files:** `packages/orchestrator/test/cost/cost-ceiling-monitor.test.ts` (new)

**Cases:**

- Task without a ceiling never emits `abort` or `warn`
- Cumulative cost exceeding ceiling emits `abort` exactly once
- Warn threshold (e.g., `warnAtPct: 80`) fires before abort
- `unregisterTask` returns the cumulative cost
- Two tasks tracked simultaneously do not interfere

**Verification:**

- `pnpm --filter @harness-engineering/orchestrator test` green

### Task D5 — Scheduler integration

**Files:** `packages/orchestrator/src/maintenance/scheduler.ts` (existing)

**Changes:**

- Before dispatch: `monitor.registerTask(taskId, definition.costCeiling)`
- Subscribe to `'abort'` event before starting the agent runner
- On abort: signal the runner to stop, set `RunResult.status = 'failure'`, `error = 'cost_ceiling_exceeded'`, `costUsd = monitor.unregisterTask(taskId)`
- On success / other failure: `costUsd = monitor.unregisterTask(taskId)`

**Verification:**

- Integration test (next task) green

### Task D6 — Integration test: scheduler aborts on cost exceed

**Files:** `packages/orchestrator/test/maintenance/scheduler-cost-ceiling.test.ts` (new)

**Cases:**

- Task with `costCeiling: { maxUsd: 0.01 }` and a mock backend reporting `inputTokens: 1_000_000` aborts with `error === 'cost_ceiling_exceeded'`
- Task without ceiling completes normally
- `RunResult.costUsd` matches the cumulative spend

**Verification:**

- `pnpm --filter @harness-engineering/orchestrator test` green

### Task D7 — Knowledge doc for cost ceiling

**Files:** `docs/knowledge/orchestrator/cost-ceiling.md` (new)

**Contents:**

- How the ceiling is enforced (turn-boundary, advisory)
- Pricing data sources and what happens when pricing is unknown
- Example config
- Trade-offs vs. hard-kill (D6 in proposal)

---

## Cross-cutting

### Task X1 — `harness.config.json` schema additions (CLI-side)

**Files:** Look for the existing JSON schema bundle; add new fields:

- `agent.backends.<name>.type: 'ssh' | 'serverless'`
- `agent.routing.isolation.{none,container,remote-sandbox}`
- `maintenance.tasks.<id>.costCeiling.{maxUsd,warnAtPct}`

**Verification:**

- `pnpm harness:validate` accepts the new fields; rejects unknown nested keys

### Task X2 — Knowledge artifact: isolation tier

**Files:** `docs/knowledge/orchestrator/dispatch-isolation.md` (new)

**Contents:**

- Explains the three tiers (`none` / `container` / `remote-sandbox`)
- Maps each existing backend to its native tier
- Walks through resolution: `routing.isolation` → backend name → optional `ContainerBackend` decoration

### Task X3 — ADRs

**Files:**

- `docs/knowledge/decisions/0013-dispatch-isolation-tier.md` — D1, D2, D8
- `docs/knowledge/decisions/0014-cost-ceiling-policy.md` — D4, D5, D6, D7

### Task X4 — CHANGELOG

**Files:** `.changeset/` — one changeset for `@harness-engineering/types` (minor), one for `@harness-engineering/orchestrator` (minor)

### Task X5 — Final `harness validate` + ship

- `pnpm harness:validate`
- `pnpm test`
- `pnpm typecheck`
- Open PR
