# ADR-001: Implement Dedicated Orchestrator Package with Multi-Tracker and Multi-Agent Support

**Date:** 2026-03-24  
**Status:** Accepted  
**Deciders:** AI Architecture Advisor, Human

---

## Context

Harness Engineering requires a long-running automation service that continuously reads work from issue trackers, creates isolated per-issue workspaces, and runs coding agent sessions for each issue inside those workspaces. The service solves four operational problems:

1. Turns issue execution into a repeatable daemon workflow instead of manual scripts.
2. Isolates agent execution in per-issue workspaces so agent commands run only inside per-issue workspace directories.
3. Keeps the workflow policy in-repo (`harness.orchestrator.md`) so teams version the agent prompt and runtime settings with their code.
4. Provides enough observability to operate and debug multiple concurrent agent runs.

We additionally require:

- **Multi-tracker support**: GitHub Issues, Linear, Jira, and local Markdown Roadmaps (e.g., `roadmap.md`).
- **Provider-agnostic agent execution**: A pluggable `AgentBackend` interface so the orchestrator can dispatch work to any compatible coding agent — not locked to a single LLM provider.

### Goals

- Poll the issue tracker on a fixed cadence and dispatch work with bounded concurrency.
- Maintain a single authoritative orchestrator state for dispatch, retries, and reconciliation.
- Create deterministic per-issue workspaces and preserve them across runs.
- Stop active runs when issue state changes make them ineligible.
- Recover from transient failures with exponential backoff.
- Load runtime behavior from a repository-owned `harness.orchestrator.md` contract.
- Expose operator-visible observability (at minimum structured logs).
- Support restart recovery without requiring a persistent database.
- **Support swappable coding agents via a common interface**, with first-class support for Claude and extensibility for other providers.

### Non-Goals

- Rich web UI or multi-tenant control plane.
- Prescribing a specific dashboard or terminal UI implementation.
- General-purpose workflow engine or distributed job scheduler.
- Built-in business logic for how to edit tickets, PRs, or comments (that logic lives in the workflow prompt and agent tooling).
- Mandating strong sandbox controls beyond what the coding agent and host OS provide.
- Mandating a single default approval, sandbox, or operator-confirmation posture for all implementations.

### Assumptions

- **Runtime**: Node.js >= 22.x (per monorepo prerequisites). TypeScript strict mode. Uses `child_process` for subprocess management and `fs`/`chokidar` for file watching.
- **Deployment**: Single-process, single-host daemon. Running multiple orchestrator instances against the same tracker project will cause duplicate dispatches. Use process supervisors (systemd, Docker) for restart, not horizontal scaling.
- **Template Engine**: Requires a Liquid-compatible template library (e.g. `liquidjs`, MIT license). This is a new dependency not currently in the monorepo.
- **Encoding**: All configuration files, prompts, and agent protocol messages are UTF-8.
- **Network**: The orchestrator host has outbound HTTPS access to tracker APIs and (for direct-api backend) LLM provider APIs.

---

## Decision

Create a new monorepo package `@harness-engineering/orchestrator` in `packages/orchestrator/` to house all daemon logic. The package sits at the top layer of the architecture, depending on `types` and `core`, and is consumed by `cli`.

All references to any specific LLM provider or agent tool (e.g. Codex, GPT, Claude) are abstracted behind a pluggable `AgentBackend` interface, allowing the orchestrator to dispatch work to any conforming agent subprocess.

---

## Agent Backend Research

The orchestrator's agent runner communicates with coding agents over a subprocess stdio bridge (line-delimited JSON messages). The following tools and protocols support this pattern and have favorable licenses:

| Agent / Protocol                 | License                              | Subprocess/Stdio                                    | Claude Support    | Notes                                                                                                             |
| -------------------------------- | ------------------------------------ | --------------------------------------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------- |
| **Claude Code SDK**              | Anthropic License (free for dev use) | ✅ Spawns CLI as subprocess, JSON over stdin/stdout | ✅ Native         | Official Anthropic SDK. Supports headless mode (`-p`), tool use, session management. Primary recommended backend. |
| **Agent Client Protocol (ACP)**  | Apache 2.0                           | ✅ JSON-RPC 2.0 over stdio                          | ✅ Via adapter    | Open standard for editor↔agent communication. Reference impl: Gemini CLI. Provider-agnostic by design.            |
| **Gemini CLI**                   | Apache 2.0                           | ✅ Native ACP/stdio                                 | ❌ Gemini only    | Google's CLI agent. Could serve as a secondary backend for Gemini models.                                         |
| **OpenCode**                     | MIT                                  | ✅ Supports MCP over stdio                          | ✅ Multi-provider | Go-based TUI agent. Supports Claude, GPT, Gemini, local models. Strong candidate for a generic backend.           |
| **Aider**                        | Apache 2.0                           | ✅ CLI invocation, structured output                | ✅ Multi-provider | Git-native coding agent. Supports Claude, GPT, Gemini, local models. Mature ecosystem.                            |
| **Claude API (Direct)**          | Anthropic API TOS                    | N/A (HTTP API)                                      | ✅ Native         | Direct Messages API with tool use. Could power a custom thin agent runner without external CLI deps.              |
| **Model Context Protocol (MCP)** | Open standard                        | ✅ Stdio transport                                  | ✅ Via agent      | Universal tool integration protocol. Agents consume MCP servers; the orchestrator can expose tools via MCP.       |

### Recommended Strategy

1. **Primary backend: Claude Code SDK** — subprocess communication via stdin/stdout JSON, headless mode (`claude -p --output-format json`), supports tool use, sandbox, and session management natively. Best integration path for Claude.
2. **Secondary backend: Aider** — Apache 2.0 licensed, supports Claude as a model provider, Git-native workflows, widely adopted. Provides an alternative subprocess-based agent that can also use Claude models.
3. **Tertiary / future: ACP-compatible agents** — Any agent implementing the Agent Client Protocol can be plugged in. This covers Gemini CLI and future ACP-compliant agents.
4. **Direct API backend (optional)**: A thin custom agent runner that calls the Claude Messages API directly with tool use, for maximum control and minimal external dependencies.

All backends conform to a common `AgentBackend` interface that the orchestrator calls uniformly.

---

## Alternatives Considered

### Core Library Expansion (`packages/core`)

_Rejected:_ Dilutes `core` as a pure logic library by introducing daemon/runner side effects. Violates the layered architectural constraint.

### Converged Daemon via `mcp-server`

_Rejected:_ Reverses the client-server hierarchy. MCP serves agents; the Orchestrator manages agents. Conflicting lifecycles.

---

## Consequences

### Positive

- Strict separation of concerns at the top architectural layer.
- Modular `IssueTrackerClient` enables multi-tracker expansion without rewriting daemon logic.
- Pluggable `AgentBackend` interface means the orchestrator is not locked to any single provider, model, or tool.
- Agent failures, stale state, and workspace isolation governed within a single-purpose state machine.

### Negative

- Scaffolding overhead for a new Turborepo package.
- Multi-tracker and multi-agent API connectors introduce testing surface area.
- Backend-specific protocol nuances may require per-adapter quirks handling.

### Neutral

- CLI gains a new daemon entrypoint (`harness orchestrator run`).

---

## Detailed Execution Specification

### 1. Main Components

| #   | Component                     | Responsibility                                                                                                                                                                               |
| --- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1   | **Workflow Loader**           | Reads `harness.orchestrator.md`, parses YAML front matter and prompt body, returns `{config, prompt_template}`                                                                               |
| 2   | **Config Layer**              | Typed getters for workflow config values, applies defaults and env var indirection, performs validation                                                                                      |
| 3   | **Issue Tracker Client**      | Fetches candidate issues in active states, fetches current states for reconciliation, fetches terminal-state issues for startup cleanup, normalizes tracker payloads into stable issue model |
| 4   | **Orchestrator**              | Owns poll tick, in-memory runtime state, dispatch/retry/stop/release decisions, session metrics and retry queue state                                                                        |
| 5   | **Workspace Manager**         | Maps issue identifiers to workspace paths, ensures directories exist, runs workspace lifecycle hooks, cleans workspaces for terminal issues                                                  |
| 6   | **Agent Runner**              | Creates workspace, builds prompt, delegates to configured `AgentBackend`, streams agent updates back to orchestrator                                                                         |
| 7   | **AgentBackend** (pluggable)  | Abstraction over the coding agent subprocess. Implementations: `ClaudeBackend`, `AiderBackend`, `ACPBackend`, `DirectAPIBackend`                                                             |
| 8   | **Status Surface** (optional) | Human-readable runtime status (terminal, dashboard, etc.)                                                                                                                                    |
| 9   | **Logging**                   | Emits structured runtime logs to configured sinks                                                                                                                                            |

### 2. Abstraction Layers

1. **Policy Layer** (repo-defined) — `harness.orchestrator.md` prompt body and team-specific rules.
2. **Configuration Layer** (typed getters) — Front matter parsing, defaults, env tokens, path normalization.
3. **Coordination Layer** (orchestrator) — Polling loop, issue eligibility, concurrency, retries, reconciliation.
4. **Execution Layer** (workspace + agent subprocess) — Filesystem lifecycle, workspace preparation, `AgentBackend` protocol.
5. **Integration Layer** (tracker adapters) — API calls and normalization for GitHub, Linear, Jira, and local Roadmap parsing.
6. **Observability Layer** (logs + optional status surface) — Operator visibility.

---

### 3. Pluggable AgentBackend Interface

```typescript
interface AgentBackend {
  /** Human-readable name for logging/status */
  readonly name: string;

  /** Start a new agent session in the given workspace */
  startSession(params: SessionStartParams): Promise<Result<AgentSession, AgentError>>;

  /** Send a turn (prompt) to an active session, yielding streamed events */
  runTurn(session: AgentSession, params: TurnParams): AsyncGenerator<AgentEvent, TurnResult, void>;

  /** Stop the agent session gracefully */
  stopSession(session: AgentSession): Promise<Result<void, AgentError>>;

  /** Check if the backend is available and healthy */
  healthCheck(): Promise<Result<void, AgentError>>;
}

interface SessionStartParams {
  workspacePath: string; // Absolute path to per-issue workspace
  permissionMode: string; // Backend-specific permission mode (e.g. 'acceptEdits', 'bypassPermissions')
  allowedTools?: string[]; // Tools the agent may use (e.g. ['Read', 'Edit', 'Bash', 'Glob', 'Grep'])
  systemPrompt?: string; // Optional system prompt override
}

interface TurnParams {
  sessionId: string; // Session ID from previous turn (for continuations)
  prompt: string; // Rendered prompt text
  isContinuation: boolean; // true for follow-up turns on same session
}

interface AgentEvent {
  type: string; // Message type (e.g. 'assistant', 'result', 'system')
  timestamp: string; // UTC ISO-8601
  subtype?: string; // Event subtype (e.g. 'success', 'error_max_turns')
  usage?: TokenUsage; // Token counts if available
  content?: unknown; // Backend-specific payload (message blocks, tool calls)
  sessionId?: string; // Session ID (available on result messages)
}
```

#### Backend Implementations

**`ClaudeBackend`** (Primary — Recommended)

- Uses the official **Claude Agent SDK** (`@anthropic-ai/claude-agent-sdk`) via the `query()` async generator.
- The SDK internally spawns the Claude Code CLI as a subprocess and communicates via NDJSON over stdin/stdout.
- **Session management**: `query()` returns typed messages (`AssistantMessage`, `ResultMessage`, `SystemMessage`). Session IDs are captured from `ResultMessage.session_id`.
- **Continuation turns**: Pass `resume: sessionId` in subsequent `query()` calls to continue an existing session with full prior context.
- **Permission modes**: `permissionMode: 'acceptEdits'` (auto-approve file changes), `permissionMode: 'bypassPermissions'` (full auto-approve for daemon use).
- **Tool control**: `allowedTools: ['Read', 'Edit', 'Bash', 'Glob', 'Grep', 'Write']`.
- Reads `ANTHROPIC_API_KEY` from environment.
- Dependency: `@anthropic-ai/claude-agent-sdk` (npm).
- License: Anthropic License (free for development).

**`AiderBackend`**

- Spawns `aider` CLI as subprocess with `--message` (single-task), `--yes` (auto-confirm), `--no-color` (prevent ANSI log pollution), and `--model` flag.
- Supports Claude (`--model claude-3.5-sonnet`), GPT, Gemini, and local models.
- Git-native: automatically commits changes.
- **Limitation**: No structured JSON output protocol. Parse exit codes and git log for result status.
- License: Apache 2.0.

**`ACPBackend`**

- Speaks Agent Client Protocol (JSON-RPC 2.0 over stdio).
- Compatible with any ACP-compliant agent (Gemini CLI, future agents).
- License: Apache 2.0 (protocol); agent-specific.

**`DirectAPIBackend`** (Optional)

- Calls the Claude Messages API directly via HTTP.
- Implements tool use loop internally (file read/write, bash execution).
- No external CLI dependency — maximum control.
- Requires `ANTHROPIC_API_KEY`.
- License: Custom implementation (our code).

#### Backend Selection Config

```yaml
# harness.orchestrator.md front matter
agent:
  backend: claude # claude | aider | acp | direct-api
  command: claude -p # Override launch command
  model: claude-sonnet-4-20250514 # Model identifier (backend-specific)
  api_key: $ANTHROPIC_API_KEY
```

---

### 4. Core Domain Model

#### 4.1 Issue

Normalized issue record used by orchestration, prompt rendering, and observability.

| Field         | Type              | Notes                                                                                                          |
| ------------- | ----------------- | -------------------------------------------------------------------------------------------------------------- |
| `id`          | string            | Stable tracker-internal ID                                                                                     |
| `identifier`  | string            | Human-readable key (e.g. `ABC-123`)                                                                            |
| `title`       | string            |                                                                                                                |
| `description` | string \| null    |                                                                                                                |
| `priority`    | integer \| null   | Lower = higher priority (1=Urgent). `0` (No priority) normalized to `null` to sort last.                       |
| `state`       | string            | Current tracker state name                                                                                     |
| `branch_name` | string \| null    | Tracker-provided branch metadata                                                                               |
| `url`         | string \| null    |                                                                                                                |
| `labels`      | string[]          | Normalized to lowercase                                                                                        |
| `blocked_by`  | BlockerRef[]      | Each: `{id, identifier, state}` (all nullable). Derived from inverse relations where relation type is `blocks` |
| `created_at`  | timestamp \| null | Parse ISO-8601                                                                                                 |
| `updated_at`  | timestamp \| null | Parse ISO-8601                                                                                                 |

#### 4.2 Workflow Definition

Parsed `harness.orchestrator.md` payload:

- `config` — YAML front matter root object.
- `prompt_template` — Markdown body after front matter, trimmed.

#### 4.3 Run Attempt

One execution attempt for one issue:

- `issue_id`, `issue_identifier`, `attempt` (null for first, ≥1 for retries), `workspace_path`, `started_at`, `status`, `error` (optional).

#### 4.4 Live Session (Agent Session Metadata)

State tracked while a coding-agent subprocess is running:

- `session_id` — Backend-assigned session identifier (e.g. from `ResultMessage.session_id` for ClaudeBackend). Used for session resume/continuation.
- `backend_name` — Which `AgentBackend` implementation is running (e.g. `claude`, `aider`, `acp`).
- `agent_pid` — Subprocess PID (if available; SDK-managed backends may not expose this directly).
- `started_at` — Timestamp when session was started.
- `last_event`, `last_timestamp`, `last_message` — Most recent event tracking for stall detection.
- `input_tokens`, `output_tokens`, `total_tokens` — Cumulative token counters.
- `last_reported_input_tokens`, `last_reported_output_tokens`, `last_reported_total_tokens` — Snapshot for delta tracking.
- `turn_count` — Number of turns started within the current worker lifetime.

#### 4.5 Retry Entry

- `issue_id`, `identifier`, `attempt` (1-based), `due_at_ms` (monotonic), `timer_handle`, `error`.

#### 4.6 Orchestrator Runtime State

Single authoritative in-memory state:

- `poll_interval_ms`, `max_concurrent_agents`
- `running` (map `issue_id` → running entry)
- `claimed` (set of issue IDs reserved/running/retrying)
- `retry_attempts` (map `issue_id` → RetryEntry)
- `completed` (set of issue IDs; bookkeeping only, not dispatch gating)
- `token_totals` (`input_tokens`, `output_tokens`, `total_tokens`, `seconds_running`)
- `rate_limits` (latest rate-limit snapshot)

#### 4.7 Stable Identifiers and Normalization Rules

- **Issue ID**: Use for tracker lookups and internal map keys.
- **Issue Identifier**: Use for logs and workspace naming.
- **Workspace Key**: Replace any char not in `[A-Za-z0-9._-]` with `_`.
- **Normalized Issue State**: Compare after `toLowerCase()`.
- **Session ID**: Backend-assigned. For ClaudeBackend, captured from `ResultMessage.session_id`. For stdio-based backends, may be derived from protocol response fields.

---

### 5. Workflow Specification (`harness.orchestrator.md`)

#### 5.1 File Discovery and Path Resolution

1. Explicit application/runtime setting (set by CLI startup path via positional argument).
2. Default: `harness.orchestrator.md` in process working directory.
3. If file cannot be read, return `missing_workflow_file` error.

#### 5.2 File Format

- If file starts with `---`, parse until next `---` as YAML front matter.
- Remaining lines become prompt body, trimmed.
- If front matter absent, entire file is prompt body; config is empty map.
- YAML front matter must decode to a map/object; non-map is an error.

#### 5.3 Front Matter Schema

**Top-level keys:** `tracker`, `polling`, `workspace`, `hooks`, `agent`. Unknown keys ignored for forward compatibility.

##### `tracker` (object)

| Field             | Type     | Default                                                    | Notes                                                                            |
| ----------------- | -------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------- |
| `kind`            | string   | —                                                          | Required for dispatch. Values: `linear`, `github`, `jira`, `roadmap`             |
| `endpoint`        | string   | Varies by kind                                             | Tracker API endpoint (Linear default: `https://api.linear.app/graphql`)          |
| `api_key`         | string   | —                                                          | Literal or `$VAR_NAME`. If `$VAR` resolves to empty, treat as missing            |
| `project_slug`    | string   | —                                                          | Required when `kind=linear`                                                      |
| `file_path`       | string   | —                                                          | Absolute or relative path to a local markdown file. Required when `kind=roadmap` |
| `active_states`   | string[] | `["Todo", "In Progress"]`                                  | Ignored by `roadmap` adapter (uses `[ ]`, `[/]`)                                 |
| `terminal_states` | string[] | `["Closed", "Cancelled", "Canceled", "Duplicate", "Done"]` | Ignored by `roadmap` adapter (uses `[x]`, `[-]`)                                 |

##### `polling` (object)

| Field         | Type                      | Default | Notes                                  |
| ------------- | ------------------------- | ------- | -------------------------------------- |
| `interval_ms` | integer \| string integer | `30000` | Dynamic reload applies to future ticks |

##### `workspace` (object)

| Field  | Type           | Default                                 | Notes                                  |
| ------ | -------------- | --------------------------------------- | -------------------------------------- |
| `root` | path \| `$VAR` | `<system-temp>/orchestrator_workspaces` | `~` expanded; path separators expanded |

##### `hooks` (object)

| Field           | Type                           | Default | Notes                                                                     |
| --------------- | ------------------------------ | ------- | ------------------------------------------------------------------------- |
| `after_create`  | multiline shell string \| null | null    | Runs only when workspace newly created. Failure aborts workspace creation |
| `before_run`    | multiline shell string \| null | null    | Runs before each agent attempt. Failure aborts current attempt            |
| `after_run`     | multiline shell string \| null | null    | Runs after each attempt. Failure logged and ignored                       |
| `before_remove` | multiline shell string \| null | null    | Runs before workspace deletion. Failure logged, cleanup proceeds          |
| `timeout_ms`    | integer                        | `60000` | Non-positive values fall back to default. Dynamic reload applies          |

##### `agent` (object)

| Field                            | Type                       | Default                  | Notes                                                                               |
| -------------------------------- | -------------------------- | ------------------------ | ----------------------------------------------------------------------------------- |
| `backend`                        | string                     | `claude`                 | Which `AgentBackend` to use: `claude`, `aider`, `acp`, `direct-api`                 |
| `command`                        | shell command string       | Backend-specific default | Override CLI command for subprocess backends (e.g. Aider). Ignored by SDK backends. |
| `model`                          | string                     | Backend-specific default | LLM model identifier passed to the backend                                          |
| `api_key`                        | string \| `$VAR`           | —                        | API key for the LLM provider                                                        |
| `max_concurrent_agents`          | integer \| string integer  | `10`                     | Dynamic reload applies                                                              |
| `max_turns`                      | integer                    | `20`                     | Max turns per worker session                                                        |
| `max_retry_backoff_ms`           | integer \| string integer  | `300000` (5 min)         | Dynamic reload applies                                                              |
| `max_concurrent_agents_by_state` | map (state → positive int) | `{}`                     | Keys normalized lowercase; invalid entries ignored                                  |
| `approval_policy`                | string                     | implementation-defined   | Passed through to agent backend                                                     |
| `sandbox_policy`                 | string                     | implementation-defined   | Passed through to agent backend                                                     |
| `turn_timeout_ms`                | integer                    | `3600000` (1 hour)       | Total turn stream timeout                                                           |
| `read_timeout_ms`                | integer                    | `5000`                   | Startup handshake request/response timeout                                          |
| `stall_timeout_ms`               | integer                    | `300000` (5 min)         | Orchestrator stall detection. If ≤ 0, disabled                                      |

##### `server` (object, extension)

| Field  | Type            | Default         | Notes                                                                      |
| ------ | --------------- | --------------- | -------------------------------------------------------------------------- |
| `port` | integer \| null | null (disabled) | Enables optional HTTP server. `0` = ephemeral port. CLI `--port` overrides |

#### 5.4 Configuration Precedence

1. Workflow file path selection (runtime setting → cwd default).
2. YAML front matter values.
3. Environment indirection via `$VAR_NAME`.
4. Built-in defaults.

Value coercion: `~` home expansion and `$VAR` expansion for path fields only; do not rewrite URIs or arbitrary shell commands.

#### 5.5 Dynamic Reload

- Watch `harness.orchestrator.md` for changes.
- Re-read and re-apply config and prompt template without restart.
- Reloaded config applies to: future dispatch, retry scheduling, reconciliation decisions, hook execution, agent launches.
- In-flight agent sessions are NOT required to restart on config change.
- Invalid reloads: do NOT crash; keep operating with last known good config; emit operator-visible error.
- Defensively re-validate during runtime operations (before dispatch) in case watch events are missed.

#### 5.6 Dispatch Preflight Validation

**Startup validation** — fail startup if:

- Workflow file cannot be loaded/parsed.
- `tracker.kind` not present or not supported.
- `tracker.api_key` not present after `$` resolution.
- `tracker.project_slug` not present when required.
- `agent.backend` not present or not supported.
- `agent.command` not resolvable (backend has no default and no override given).

**Per-tick validation** — if fails: skip dispatch for that tick, keep reconciliation active, emit error.

#### 5.7 Prompt Template Contract

- **Engine**: Strict template engine (Liquid-compatible semantics sufficient).
- **Variables**: `issue` (full normalized object including labels/blockers) and `attempt` (null on first, integer on retry/continuation).
- **Strict mode**: Unknown variables MUST fail rendering. Unknown filters MUST fail rendering.
- **Fallback**: If prompt body is empty, MAY use minimal default (`"You are working on an issue."`). File read/parse failures are NOT silently fallen back.
- **First turn**: Uses full rendered task prompt.
- **Continuation turns**: Sends only continuation guidance to existing thread, NOT the original task prompt already in thread history.

#### 5.8 Workflow Validation Errors

- `missing_workflow_file`
- `workflow_parse_error`
- `workflow_front_matter_not_a_map`
- `template_parse_error` (during rendering)
- `template_render_error` (unknown variable/filter)
- `unsupported_agent_backend`

Dispatch gating: Workflow file errors block new dispatches. Template errors fail only the affected run attempt.

---

### 6. Orchestration State Machine

#### 6.1 Issue Orchestration States (Internal, not tracker states)

1. **Unclaimed** — Not running, no retry scheduled.
2. **Claimed** — Reserved to prevent duplicate dispatch. In practice, either Running or RetryQueued.
3. **Running** — Worker task exists in `running` map.
4. **RetryQueued** — Worker not running, retry timer exists in `retry_attempts`.
5. **Released** — Claim removed (terminal, non-active, missing, or retry path completed).

**Multi-turn nuance:**

- A successful worker exit does not mean the issue is done forever.
- The worker may continue through multiple back-to-back agent turns before exiting.
- After each normal turn completion, the worker re-checks the tracker issue state.
- If still active, the worker starts another turn on the same live thread in the same workspace, up to `agent.max_turns`.
- Once the worker exits normally, the orchestrator schedules a short continuation retry (~1s) to re-check whether the issue remains active.

#### 6.2 Run Attempt Lifecycle Phases

1. `PreparingWorkspace`
2. `BuildingPrompt`
3. `LaunchingAgent`
4. `InitializingSession`
5. `StreamingTurn`
6. `Finishing`
7. `Succeeded`
8. `Failed`
9. `TimedOut`
10. `Stalled`
11. `CanceledByReconciliation`

#### 6.3 Transition Triggers

- **Poll Tick**: Reconcile → Validate → Fetch → Sort → Dispatch.
- **Worker Exit (normal)**: Remove running entry → update totals → schedule continuation retry (attempt `1`).
- **Worker Exit (abnormal)**: Remove running entry → update totals → schedule exponential-backoff retry.
- **Agent Update Event**: Update live session fields, token counters, rate limits.
- **Retry Timer Fired**: Re-fetch active candidates → re-dispatch or release claim.
- **Reconciliation State Refresh**: Stop runs whose tracker states are terminal or non-active.
- **Stall Timeout**: Kill worker → schedule retry.

#### 6.4 Idempotency and Recovery Rules

- Orchestrator serializes all state mutations through one authority.
- `claimed` and `running` checks required before launching any worker.
- Reconciliation runs before dispatch on every tick.
- Restart recovery is tracker-driven and filesystem-driven (no durable DB).
- Startup terminal cleanup removes stale workspaces.

---

### 7. Polling, Scheduling, and Reconciliation

#### 7.1 Poll Loop

At startup: validate config → perform startup cleanup → schedule immediate tick → repeat every `polling.interval_ms`.

**Tick sequence:**

1. Reconcile running issues.
2. Run dispatch preflight validation.
3. Fetch candidate issues from tracker using active states.
4. Sort issues by dispatch priority.
5. Dispatch eligible issues while slots remain.
6. Notify observability/status consumers.

If per-tick validation fails: skip dispatch, but reconciliation still happens first.

#### 7.2 Candidate Selection Rules

An issue is dispatch-eligible only if ALL are true:

- Has `id`, `identifier`, `title`, and `state`.
- State is in `active_states` AND NOT in `terminal_states`.
- NOT already in `running`.
- NOT already in `claimed`.
- Global concurrency slots available.
- Per-state concurrency slots available.
- **Blocker rule for `Todo` state**: Do NOT dispatch when any blocker is non-terminal.

**Sort order (stable):**

1. `priority` ascending (1..4 preferred; null/unknown sorts last).
2. `created_at` oldest first.
3. `identifier` lexicographic tie-breaker.

#### 7.3 Concurrency Control

- **Global**: `available_slots = max(max_concurrent_agents - running_count, 0)`.
- **Per-state**: `max_concurrent_agents_by_state[state]` if present (key normalized lowercase), otherwise global limit.
- **Optional SSH per-host limit**: `worker.max_concurrent_agents_per_host`.

#### 7.4 Retry and Backoff

- Cancel any existing retry timer for same issue before creating new one.
- **Continuation retries** (normal worker exit): Fixed delay `1000ms`.
- **Failure retries**: `delay = min(10000 * 2^(attempt - 1), agent.max_retry_backoff_ms)`.
- Power capped by `max_retry_backoff_ms` (default 300000 / 5 min).
- **No max attempt limit** (by design): Retries continue indefinitely until the tracker issue state changes to terminal/non-active. This matches the Symphony spec's intentional design — the tracker is the sole authority for stopping retries. If an issue is failing repeatedly, the operator changes its tracker state.

**Retry handling:**

1. Fetch active candidate issues.
2. Find issue by `issue_id`.
3. If not found → release claim.
4. If found and still eligible → dispatch if slots available, else requeue with error `"no available orchestrator slots"`.
5. If found but no longer active → release claim.

#### 7.5 Active Run Reconciliation (every tick)

**Part A: Stall detection**

- For each running issue, compute `elapsed_ms` since `last_timestamp` (or `started_at` if no events).
- If `elapsed_ms > agent.stall_timeout_ms` → terminate worker → queue retry.
- If `stall_timeout_ms <= 0` → skip stall detection entirely.

**Part B: Tracker state refresh**

- Fetch current states for all running issue IDs.
- Terminal state → terminate worker AND clean workspace.
- Still active → update in-memory issue snapshot.
- Neither active nor terminal → terminate worker WITHOUT workspace cleanup.
- If state refresh fails → keep workers running, retry next tick.

#### 7.6 Startup Terminal Workspace Cleanup

1. Query tracker for issues in terminal states.
2. Remove corresponding workspace directories.
3. If fetch fails → log warning → continue startup.

---

### 8. Workspace Management and Safety

#### 8.1 Layout

- Root: `workspace.root` (normalized path).
- Per-issue: `<workspace.root>/<sanitized_issue_identifier>`.
- Workspaces are reused across runs for the same issue.
- Successful runs do NOT auto-delete workspaces.

#### 8.2 Creation and Reuse Algorithm

1. Sanitize identifier to `workspace_key` (`[A-Za-z0-9._-]` only, others become `_`).
2. **Truncate** `workspace_key` to a maximum of 64 characters to prevent `ENAMETOOLONG` OS errors for verbose roadmap task titles.
3. Compute path under workspace root.
4. Ensure path exists as directory.
5. Mark `created_now=true` only if directory was created during this call.
6. If `created_now=true` → run `after_create` hook if configured.

#### 8.3 Optional Workspace Population

Implementation-defined (typically via hooks). Failure returns error for current attempt. New workspaces may be removed on population failure. Reused workspaces should NOT be destructively reset unless explicitly chosen.

#### 8.4 Hook Execution Contract

- Execute via `sh -lc <script>` (or `bash -lc`) with workspace dir as `cwd`.
- Timeout: `hooks.timeout_ms` (default 60000).
- Log hook start, failures, and timeouts.

| Hook                            | Fatal?                       |
| ------------------------------- | ---------------------------- |
| `after_create` failure/timeout  | Fatal to workspace creation  |
| `before_run` failure/timeout    | Fatal to current run attempt |
| `after_run` failure/timeout     | Logged and ignored           |
| `before_remove` failure/timeout | Logged and ignored           |

#### 8.5 Disk Exhaustion

- If workspace directory creation fails due to `ENOSPC` (disk full), fail the current run attempt.
- Log a high-severity warning with `workspace.root` path.
- Operators should configure workspace cleanup policies via hooks or external monitoring.

#### 8.6 Safety Invariants (CRITICAL)

1. **Run agent ONLY in per-issue workspace path.** Validate `cwd === workspace_path` before launch.
2. **Workspace path MUST stay inside workspace root.** Normalize both to absolute. Reject any path outside root.
3. **Workspace key is sanitized.** Only `[A-Za-z0-9._-]` allowed; replace all others with `_`.

---

### 9. Agent Runner Protocol (Provider-Agnostic)

#### 9.1 Launch Contract

- Command: `agent.command` (backend-specific default if not overridden).
- Invocation: `bash -lc <agent.command>`.
- Working directory: workspace path.
- Stdout/stderr: separate streams. Protocol messages on stdout only.
- Recommended max line size: 10 MB.

#### 9.2 Session Lifecycle

The `AgentBackend` interface abstracts the underlying protocol. Regardless of backend:

1. **Start Session** — Initialize the agent subprocess in the workspace directory with configured policies.
2. **Run Turn** — Send a rendered prompt (or continuation guidance) and stream events back to the orchestrator.
3. **Stop Session** — Gracefully terminate the agent subprocess.

Session identifiers:

- For **SDK-based backends** (ClaudeBackend): Read `session_id` from `ResultMessage.session_id`. Use `resume: sessionId` for continuation turns.
- For **stdio-based backends** (ACPBackend): Read `thread_id` from protocol response. Construct `session_id = "<thread_id>-<turn_id>"`. Reuse `thread_id` for continuation turns.
- For **CLI-based backends** (AiderBackend): Session is the subprocess lifetime. Continuation turns are separate process invocations. Session ID derived from issue identifier + attempt.

#### 9.3 Streaming Turn Processing

Read events from the agent until the turn terminates:

- Turn completed → success.
- Turn failed → failure.
- Turn cancelled → failure.
- Turn timeout (`agent.turn_timeout_ms`) → failure.
- Subprocess exit → failure.

**Continuation:** If the worker decides to continue after a successful turn, issue another turn on the same live thread. Agent subprocess remains alive across continuation turns; stopped only when worker run is ending.

**Line handling (for stdio-based backends):**

- Read protocol messages from stdout only.
- Buffer partial stdout lines until newline arrives.
- Attempt JSON parse on complete stdout lines.
- Stderr: ignore or log as diagnostics; do NOT attempt protocol parsing.

#### 9.4 Emitted Runtime Events

Events emitted upstream to orchestrator include: `session_started`, `startup_failed`, `turn_completed`, `turn_failed`, `turn_cancelled`, `turn_ended_with_error`, `turn_input_required`, `approval_auto_approved`, `unsupported_tool_call`, `notification`, `other_message`, `malformed`.

Each event includes: `event`, `timestamp` (UTC), `agent_pid`, optional `usage` map.

#### 9.5 Approval, Tool Calls, and User Input Policy

- Implementation-defined per backend; must document chosen posture.
- Approval/user-input requests must NOT stall indefinitely.
- **User input required → HARD failure** (daemon is non-interactive).
- Unsupported dynamic tool calls: return tool failure response and continue session.

**Optional Client-Side Tool Extension: `linear_graphql`**

- If implemented, the orchestrator exposes a `linear_graphql` tool to the agent session.
- Purpose: Execute raw GraphQL against Linear using the orchestrator's configured auth.
- Input: `{ "query": "...", "variables": {...} }`. Single operation only.
- Execution: Reuses the active `tracker.endpoint` and `tracker.api_key`.
- Result semantics: Transport success = tool success. GraphQL execution errors (`errors` array) return `success=false` but preserve the GraphQL body for the agent to inspect.

**Optional Client-Side Tool Extension: `update_roadmap_task`**

- Required if `tracker.kind = 'roadmap'` is implemented, allowing isolated agents to update the true roadmap file.
- Purpose: Modify the state (checkbox) of a roadmap item in the host's real `roadmap.md`.
- Input: `{ "task_id": "md5-hash-id", "status": "done" | "in_progress" | "todo" }`.
- Execution: Orchestrator reads `roadmap.md`, finds the line by hash, updates the checkbox (`[x]`, `[/]`, `[ ]`), and writes back to disk.
- Result semantics: `success=true` if file updated. `success=false` if ID not found.

#### 9.6 Timeouts and Error Mapping

| Timeout                         | Field                    | Default   |
| ------------------------------- | ------------------------ | --------- |
| Startup request/response        | `agent.read_timeout_ms`  | `5000`    |
| Total turn stream               | `agent.turn_timeout_ms`  | `3600000` |
| Event inactivity (orchestrator) | `agent.stall_timeout_ms` | `300000`  |

Normalized error categories: `agent_not_found`, `invalid_workspace_cwd`, `response_timeout`, `turn_timeout`, `process_exit`, `response_error`, `turn_failed`, `turn_cancelled`, `turn_input_required`.

#### 9.7 Worker Attempt Algorithm (Reference)

```
run_agent_attempt(issue, attempt, backend):
  workspace = create_for_issue(issue.identifier)
  if failed → fail_worker("workspace error")

  if run_hook("before_run", workspace.path) failed → fail_worker("before_run hook error")

  session = backend.startSession({workspacePath: workspace.path, ...policies})
  if failed → run_hook_best_effort("after_run") → fail_worker("agent session startup error")

  max_turns = config.agent.max_turns
  turn_number = 1

  while true:
    prompt = build_turn_prompt(template, issue, attempt, turn_number, max_turns)
    if failed → backend.stopSession(session) → run_hook_best_effort("after_run") → fail_worker("prompt error")

    for await event of backend.runTurn(session, {prompt, isContinuation: turn_number > 1, sessionId: session.id}):
      forward event to orchestrator (update tokens, stall timestamp, etc.)
    turn_result = final result from generator
    if failed → backend.stopSession(session) → run_hook_best_effort("after_run") → fail_worker("agent turn error")

    refreshed = tracker.fetch_issue_states_by_ids([issue.id])
    if failed → backend.stopSession(session) → run_hook_best_effort("after_run") → fail_worker("issue state refresh error")

    issue = refreshed[0] or issue
    if issue.state not active → break
    if turn_number >= max_turns → break
    turn_number++

  backend.stopSession(session) → run_hook_best_effort("after_run") → exit_normal()
```

#### 9.8 Worker Exit and Retry Handling (Reference)

```
on_worker_exit(issue_id, reason, state):
  running_entry = state.running.remove(issue_id)
  state = add_runtime_seconds_to_totals(state, running_entry)

  if reason == normal:
    state.completed.add(issue_id)  # bookkeeping only
    schedule_retry(state, issue_id, 1, {identifier, delay_type: continuation})  # 1000ms
  else:
    schedule_retry(state, issue_id, next_attempt, {identifier, error: "worker exited: <reason>"})

  notify_observers()
```

```
on_retry_timer(issue_id, state):
  retry_entry = state.retry_attempts.pop(issue_id)
  if missing → return

  candidates = tracker.fetch_candidate_issues()
  if failed → schedule_retry(issue_id, attempt + 1, {error: "retry poll failed"})

  issue = find_by_id(candidates, issue_id)
  if null → state.claimed.remove(issue_id) → return
  if no_available_slots → schedule_retry(issue_id, attempt + 1, {error: "no available orchestrator slots"})

  dispatch_issue(issue, state, attempt=retry_entry.attempt)
```

---

### 10. Issue Tracker Integration Contract

#### 10.1 Required Operations

1. `fetch_candidate_issues()` — Active state issues for configured project.
2. `fetch_issues_by_states(state_names)` — For startup terminal cleanup.
3. `fetch_issue_states_by_ids(issue_ids)` — For active-run reconciliation.

#### 10.2 Tracker Adapters

**Linear Adapter:**

- GraphQL endpoint (default `https://api.linear.app/graphql`).
- Auth token in `Authorization` header.
- `project_slug` maps to Linear `slugId`.
- **Candidate query:** Filters projects using `project: { slugId: { eq: $projectSlug } }`. Needs pagination. Page size default `50`. Network timeout: `30000ms`.
- **State refresh query:** Uses GraphQL issue IDs with variable type `[ID!]`.

**GitHub Issues Adapter:**

- REST API v3 or GraphQL v4.
- Map `open`/`closed` to active/terminal states. Filter by labels, milestones, or projects.
- Pagination via Link headers or cursor.
- Auth via `$GITHUB_TOKEN`.

**Jira Adapter:**

- REST API v3.
- JQL queries for active/terminal state filtering.
- **Note**: No existing Jira connector code exists in the monorepo today. The `@harness-engineering/graph` package describes Jira as one of its external connectors, but implementation status must be verified before depending on it. Plan as a from-scratch implementation unless confirmed otherwise.
- Auth via `$JIRA_API_TOKEN` + `$JIRA_EMAIL`.

**Roadmap Adapter (Local Markdown):**

- Reads a local Markdown file specified by `tracker.file_path` (e.g., `docs/roadmap.md`).
- Parses Github-Flavored Markdown task lists: `- [ ]`, `* [x]`, etc.
- **State mapping**: `[ ]` (empty) maps to "Todo". `[/]` or `[-]` (partial) maps to "In Progress". Both are considered active. `[x]` or `[X]` maps to "Done" (terminal).
- **ID Normalization**: Uses a deterministic hash of the sanitized task line content (e.g., `md5(task_text)`) as the stable `id`. This prevents IDs from shifting when lines are added/removed above the task.
- **Identifier**: A slugified prefix of the task content (e.g., `implement-dedicated-orche`).
- **Title/Description**: The text immediately following the checkbox.
- **Hierarchy mapping (Blockers)**: Nested lists define blockers. A parent task is considered blocked by its uncompleted child tasks. This allows the orchestrator blocker resolution rule to prevent parent task dispatch while children are active.

#### 10.3 Normalization Rules

- `labels` → lowercase strings.
- `blocked_by` → derived from inverse/blocking relations.
- `priority` → integer only; `0` or non-integers map to `null` (ensures "No priority" sorts after "Low").
- `created_at`, `updated_at` → parse ISO-8601.

#### 10.4 Error Handling

Categories: `unsupported_tracker_kind`, `missing_tracker_api_key`, `missing_tracker_project_slug`, `api_request_error`, `api_status_error`, `graphql_errors`, `unknown_payload`, `missing_end_cursor`.

Orchestrator behavior:

- Candidate fetch failure → log, skip dispatch this tick.
- Running-state refresh failure → log, keep active workers.
- Startup terminal cleanup failure → log warning, continue startup.

#### 10.5 Tracker Writes Boundary

The orchestrator is a **read-only** scheduler/runner and tracker reader. Ticket mutations (state transitions, comments, PR links) happen exclusively via the coding agent using tools in the workflow/runtime environment.

---

### 11. Prompt Construction and Context Assembly

#### 11.1 Inputs

- `workflow.prompt_template`, normalized `issue` object, optional `attempt` integer.

#### 11.2 Rendering Rules

- Render with strict variable and filter checking.
- Convert issue object keys to strings for template compatibility.
- Preserve nested arrays/maps (labels, blockers) so templates can iterate.

#### 11.3 Retry/Continuation Semantics

`attempt` is passed to the template because workflows may provide differentiated instructions for first run, continuation, or error retry.

#### 11.4 Failure Semantics

Prompt rendering failure → fail run attempt immediately → orchestrator decides retry behavior.

---

### 12. Logging, Status, and Observability

#### 12.1 Logging Conventions

Required context fields:

- Issue-related logs: `issue_id`, `issue_identifier`.
- Session lifecycle logs: `session_id`, `backend_name`.
- Stable `key=value` phrasing, action outcome, concise failure reason.
- Avoid logging large raw payloads.

#### 12.2 Logging Outputs

- Operators must see startup/validation/dispatch failures without a debugger.
- If a log sink fails, service continues and emits warning through remaining sinks.

#### 12.3 Runtime Snapshot (Optional but Recommended)

Synchronous snapshot for dashboards:

- `running` (list with `turn_count` and `backend_name` per row), `retrying`, `token_totals` (`input_tokens`, `output_tokens`, `total_tokens`, `seconds_running` including active sessions), `rate_limits`.

#### 12.4 Token Accounting Rules

- Prefer absolute thread totals when available.
- Ignore delta-style payloads for dashboard totals.
- Track deltas relative to last reported totals to avoid double-counting.
- Accumulate aggregate totals in orchestrator state.
- Runtime reported as live aggregate at snapshot time (cumulative ended + active elapsed).
- Add run duration to cumulative counter when session ends.

#### 12.5 Optional HTTP Server Extension

If implemented:

- Enabled via CLI `--port` or `server.port` in front matter.
- CLI `--port` overrides `server.port`.
- Bind loopback by default.
- Dashboard at `/` showing active sessions, retry delays, token consumption, events, health.

**JSON REST API (`/api/v1/*`):**

- `GET /api/v1/state` — Summary view of running sessions, retry queue, active/ended token and runtime totals, rate limits.
  _Example:_ `{"counts": {"running": 2, "retrying": 1}, "running": [...], "retrying": [...], "codex_totals": {"total_tokens": 7400, "seconds_running": 1834.2}}`
- `GET /api/v1/<issue_identifier>` — Issue-specific runtime/debug details. 404 if unknown. Include attempts, last_event, last_error.
- `POST /api/v1/refresh` — Queue immediate poll + reconciliation (202 Accepted).
- Unsupported methods → 405. Errors → `{"error": {"code": "...", "message": "..."}}`.

#### 12.6 Humanized Agent Event Summaries (Optional)

- If implemented, treat as observability-only output.
- Do NOT make orchestrator logic depend on humanized strings.
- Useful for terminal/dashboard display of raw agent protocol events.

#### 12.7 Terminal User Interface (TUI) Dashboard

The CLI command `harness orchestrator run` MUST provide a rich, self-updating Terminal UI (TUI) matching the reference Symphony application's layout.

**Global Header Requirements:**

- **Agents**: Active vs Max capacity (e.g., `19/50`).
- **Throughput**: Calculated tokens per second (`tps`) across all active agents.
- **Runtime**: Orchestrator uptime process duration.
- **Tokens**: Global aggregate `in`, `out`, and `total` tokens.
- **Rate Limits**: Display configured/active rate limits or quotas (if supported by backend).
- **Project**: The tracker URL configured in the workflow.
- **Next refresh**: Countdown timer to the next orchestrator poll/reconciliation tick.

**Running Agents Table Requirements:**
Displays one row per active worker with the following columns:

- `ID`: Issue tracker identifier (e.g., `MT-725`).
- `STAGE`: Current issue tracker state (e.g., `Todo`, `In Progress`, `Rework`).
- `PID`: Agent subprocess OS Process ID.
- `AGE / TURN`: Elapsed duration of the current session / Current turn number (e.g., `1m 19s / 1`).
- `TOKENS`: Aggregate tokens consumed by this specific session.
- `SESSION`: Truncated backend session ID (e.g., `019c...ac5e10`).
- `EVENT`: Truncated humanized formatting of the latest agent protocol event (e.g., `command output streaming: > npm run...`, `item started: reasoning`, `turn diff updated`).

**Backoff Queue Requirements:**
Displays a summary or table of issues currently in the retry/backoff queue. Wait states should reflect "No queued retries" when empty.

---

### 13. Failure Model and Recovery Strategy

#### 13.1 Failure Classes

1. **Workflow/Config**: Missing `harness.orchestrator.md`, invalid YAML, unsupported tracker kind, missing credentials, unsupported agent backend.
2. **Workspace**: Directory creation failure, population/sync failure, invalid path config, hook timeout/failure.
3. **Agent Session**: Startup handshake failure, turn failed/cancelled/timeout, user input requested, subprocess exit, stalled session.
4. **Tracker**: API transport errors, non-200 status, GraphQL errors, malformed payloads.
5. **Observability**: Snapshot timeout, dashboard render errors, log sink config failure.

#### 13.2 Recovery Behavior

- Dispatch validation failures → skip dispatches, keep service alive, continue reconciliation.
- Worker failures → exponential backoff retries.
- Tracker candidate-fetch failures → skip this tick, retry next.
- Reconciliation state-refresh failures → keep workers, retry next tick.
- Dashboard/log failures → do NOT crash orchestrator.

#### 13.3 Partial State Recovery (Restart)

Intentionally in-memory. After restart:

- No retry timers restored.
- No running sessions assumed recoverable.
- Recovery via: startup terminal workspace cleanup → fresh polling → re-dispatch eligible work.

#### 13.4 Operator Intervention Points

- Edit `harness.orchestrator.md` (auto-detected, re-applied without restart).
- Change issue states in tracker (terminal → stop + cleanup; non-active → stop without cleanup).
- Restart service for process recovery or deployment.

---

### 14. Security and Operational Safety

#### 14.1 Trust Boundary

Implementation-defined. Must document explicitly whether targeting trusted/restrictive environments and chosen approval/sandbox/confirmation policy.

#### 14.2 Filesystem Safety (Mandatory)

- Workspace path under configured root.
- Coding-agent cwd = per-issue workspace path.
- Sanitized directory names only.

#### 14.3 Recommended Hardening

- Dedicated OS user.
- Restricted workspace root permissions.
- Dedicated volume for workspace root.

#### 14.4 Secret Handling

- `$VAR` indirection in workflow config.
- NEVER log API tokens or secret env values.
- Validate secret presence without printing.

#### 14.5 Hook Script Safety

- Hooks are fully trusted configuration (from `harness.orchestrator.md`).
- Run inside workspace directory.
- Truncate hook output in logs.
- Timeout required to prevent orchestrator hang.

#### 14.6 Harness Hardening Guidance

- Tighten agent approval/sandbox settings per backend.
- Add external isolation (container/VM/network restrictions).
- Filter which tracker projects/labels/teams are eligible for dispatch.
- Narrow tool access available to the agent.
- Minimize credentials/paths/destinations available to agent.

---

### 15. CLI and Host Lifecycle

- CLI accepts optional positional workflow path argument (`path-to-harness.orchestrator.md`).
- Uses `./harness.orchestrator.md` when no argument provided.
- Errors on nonexistent explicit path or missing default.
- CLI surfaces startup failure cleanly.
- Exits with success on normal startup and shutdown.
- Exits nonzero on startup failure or abnormal exit.

#### 15.1 Graceful Shutdown Sequence

On `SIGTERM` or `SIGINT`:

1. Stop accepting new dispatches (disable poll tick).
2. Cancel all pending retry timers.
3. Send `SIGTERM` to all running agent subprocesses.
4. Wait up to `30` seconds for graceful agent exit.
5. Force-kill remaining agent processes after grace period.
6. Run `after_run` hooks best-effort for each terminated session.
7. Do NOT clean workspaces (preserve for restart recovery).
8. Exit `0` if all agents stopped within grace period; nonzero otherwise.

---

### 16. Config Cheat Sheet (Quick Reference)

```
tracker.kind:                     string, required (linear | github | jira | roadmap)
tracker.endpoint:                 string, default varies by kind
tracker.api_key:                  string or $VAR, required
tracker.project_slug:             string, required when kind=linear
tracker.file_path:                string, required when kind=roadmap
tracker.active_states:            string[], default ["Todo", "In Progress"]
tracker.terminal_states:          string[], default ["Closed", "Cancelled", "Canceled", "Duplicate", "Done"]
polling.interval_ms:              integer, default 30000
workspace.root:                   path, default <system-temp>/orchestrator_workspaces
hooks.after_create:               shell script or null
hooks.before_run:                 shell script or null
hooks.after_run:                  shell script or null
hooks.before_remove:              shell script or null
hooks.timeout_ms:                 integer, default 60000
agent.backend:                    string, default "claude" (claude | aider | acp | direct-api)
agent.command:                    shell command, backend-specific default
agent.model:                      string, backend-specific default
agent.api_key:                    string or $VAR
agent.max_concurrent_agents:      integer, default 10
agent.max_turns:                  integer, default 20
agent.max_retry_backoff_ms:       integer, default 300000 (5m)
agent.max_concurrent_agents_by_state: map of positive ints, default {}
agent.approval_policy:            string, implementation-defined
agent.sandbox_policy:             string, implementation-defined
agent.turn_timeout_ms:            integer, default 3600000 (1h)
agent.read_timeout_ms:            integer, default 5000
agent.stall_timeout_ms:           integer, default 300000 (5m), ≤0 disables
server.port:                      integer (extension), null = disabled
```

---

### 17. Test and Validation Matrix

#### Core Conformance (Required)

**Workflow and Config Parsing:**

- Path precedence (explicit vs cwd default).
- Dynamic watch/reload/re-apply.
- Invalid reload keeps last good config + emits error.
- Missing file → typed error. Invalid YAML → typed error. Non-map front matter → typed error.
- Defaults apply for optional values.
- `tracker.kind` validation, `$VAR` resolution, `~` expansion.
- `agent.backend` validation.
- Per-state concurrency: normalize keys, ignore invalid values.
- Strict prompt rendering with `issue` and `attempt`.

**Workspace Manager:**

- Deterministic path per identifier. Missing → create. Existing → reuse.
- Hook lifecycle (create, before_run, after_run, before_remove) with correct fatal/ignore semantics.
- Sanitization and root containment enforced before agent launch.

**Issue Tracker Client:**

- Candidate fetch uses active states and project slug.
- Pagination preserves order. Blockers from inverse relations. Labels lowercase.
- State refresh by ID.
- Roadmap adapter: hashes task strings for stable IDs, treats unchecked as active/checked as terminal, models nested tasks as blockers.
- Error mapping for all failure categories.

**Orchestrator Dispatch, Reconciliation, and Retry:**

- Sort: priority → oldest → identifier.
- `Todo` + non-terminal blockers → not eligible. `Todo` + terminal blockers → eligible.
- Active refresh updates entry. Non-active stops without cleanup. Terminal stops + cleans.
- Normal exit → continuation retry (1s). Abnormal exit → exponential backoff.
- Backoff cap uses configured `max_retry_backoff_ms`.
- Stall detection kills + retries. Slot exhaustion requeues with error.

**Agent Backend Client:**

- Launch via configured backend, startup handshake, policy payloads.
- Read/turn timeouts enforced. Partial lines buffered (stdio backends). Stdout-only protocol parsing.
- Approval/user-input handling per documented policy (no indefinite stall).
- Unsupported tool calls rejected without stalling.
- Usage/rate-limit extraction from agent events.

**Observability and TUI:**

- Validation failures operator-visible.
- Structured logging with context fields.
- Sink failures don't crash orchestration.
- Token aggregation correct across updates.
- TUI displays global headers, Running table, and Backoff queue accurately based on orchestrator state.

**CLI Lifecycle:**

- Optional positional workflow path. Default `./harness.orchestrator.md`.
- Nonexistent path → error. Startup failure → clean error + nonzero exit.

#### Extension Conformance (If Implemented)

- HTTP server: port precedence, loopback bind, baseline endpoints, error envelope.
- HTTP server port reconfiguration requires restart (live rebind not required).
- Additional agent backends: conform to `AgentBackend` interface contract.
- SSH workers: host pool scheduling, per-host cap, continuation affinity.
- Snapshot API: running/retrying/totals/rate-limits, timeout/unavailable handling.
- `linear_graphql` tool: advertises correctly, restricts to single operation, returns GraphQL bodies on error.

---

### 18. Implementation Checklist (Definition of Done)

#### Required for Conformance

- [ ] Workflow path selection (explicit + cwd default)
- [ ] `harness.orchestrator.md` loader with YAML front matter + prompt body split
- [ ] Typed config layer with defaults and `$` resolution
- [ ] Dynamic `harness.orchestrator.md` watch/reload/re-apply
- [ ] `AgentBackend` interface definition
- [ ] At least one `AgentBackend` implementation (recommended: `ClaudeBackend`)
- [ ] Polling orchestrator with single-authority mutable state
- [ ] Issue tracker client with candidate fetch + state refresh + terminal fetch
- [ ] At least one tracker adapter (recommended: Linear, Roadmap, then GitHub)
- [ ] Workspace manager with sanitized per-issue workspaces
- [ ] Workspace lifecycle hooks (all four, with timeout config)
- [ ] Agent subprocess client with backend-appropriate protocol
- [ ] Strict prompt rendering with `issue` and `attempt`
- [ ] Exponential retry queue with continuation retries after normal exit
- [ ] Configurable retry backoff cap
- [ ] Reconciliation that stops runs on terminal/non-active tracker states
- [ ] Workspace cleanup for terminal issues (startup sweep + active transition)
- [ ] Structured logs with `issue_id`, `issue_identifier`, `session_id`, `backend_name`
- [ ] Operator-visible observability via Terminal UI (TUI) Dashboard

#### Recommended Extensions

- [ ] HTTP server with dashboard + JSON API
- [ ] Additional `AgentBackend` implementations (Aider, ACP, Direct API)
- [ ] Additional tracker adapters (Jira, GitHub)
- [ ] `linear_graphql` optional client-side tool extension
- [ ] Roadmap tracker editing tool (allows agent to auto-check roadmap items in isolated workspaces)
- [ ] SSH remote worker execution
- [ ] Persistent retry queue across restarts
- [ ] First-class tracker write APIs in orchestrator

#### Operational Validation

- [ ] Real integration profile with valid credentials
- [ ] Hook execution verified on target OS/shell
- [ ] HTTP server port behavior verified on target environment

---

## Action Items

- [ ] **1. Scaffold Package** — Create `packages/orchestrator/` with `package.json`, `tsconfig.json`, barrel exports, Turborepo registration. — owner: Execution Agent
- [ ] **2. Domain Types** — Define all entities (Issue, WorkflowDefinition, RunAttempt, LiveSession, RetryEntry, OrchestratorState, AgentBackend, AgentEvent) in `packages/types/`. — owner: Execution Agent
- [ ] **3. Workflow Loader & Config** — Implement `harness.orchestrator.md` discovery, YAML parsing, typed config layer with defaults/`$VAR`/`~` expansion, file watcher for dynamic reload. — owner: Execution Agent
- [ ] **4. Prompt Engine** — Liquid-compatible strict template renderer with `issue` and `attempt` variables. — owner: Execution Agent
- [ ] **5. Workspace Manager** — Sanitization, path isolation, create/reuse, hook execution with timeouts. — owner: Execution Agent
- [ ] **6. AgentBackend Interface + ClaudeBackend** — Define `AgentBackend` and implement primary Claude Code SDK backend. — owner: Execution Agent
- [ ] **7. Additional Backends** — Implement `AiderBackend`, `ACPBackend`, and/or `DirectAPIBackend`. — owner: Execution Agent
- [ ] **8. Unified Tracker Client** — `IssueTrackerClient` interface + Linear adapter + GitHub adapter (Jira as extension). — owner: Execution Agent
- [ ] **9. Orchestration State Machine** — Poll loop, candidate selection, concurrency control, retry/backoff queue, reconciliation, stall detection. — owner: Execution Agent
- [ ] **10. Observability & TUI** — Structured logging, optional HTTP server extension, and interactive Terminal UI (TUI) matching reference spec. — owner: Execution Agent
- [ ] **11. CLI Integration** — `harness orchestrator run` command in `packages/cli/`. — owner: Execution Agent
- [ ] **12. Test Suite** — Full Core Conformance test coverage per validation matrix. — owner: Execution Agent

---

## Appendix A: SSH Worker Extension (Optional)

This appendix describes a potential extension where the orchestrator keeps one central coordinator but executes worker runs on remote hosts over SSH. This is NOT required for conformance and is deferred to a future ADR if needed.

### A.1 Execution Model

- Orchestrator remains single source of truth for polling, claims, retries, reconciliation.
- `worker.ssh_hosts` provides SSH destinations.
- `workspace.root` interpreted on remote host, not orchestrator host.
- Agent launched over SSH stdio; orchestrator still owns session lifecycle.
- Continuation turns stay on same host and workspace.

### A.2 Scheduling

- SSH hosts treated as pool for dispatch.
- Prefer previously used host on retries.
- `worker.max_concurrent_agents_per_host` as shared per-host cap.
- When all hosts at capacity, wait rather than falling back to different mode.
- Failover to another host only before meaningful side effects; otherwise treat as new attempt.
