# Smoke Test — Features across the last releases

You are running the **feature** slice of a release smoke test for the
`smithers-orchestrator` npm package. Your job: prove that every user-facing
change shipped across the last few releases actually holds against the published
package, and that the baseline install path is not broken.

You are a real coding agent with a shell. **Run the commands yourself**, capture
real exit codes and output, and judge pass/fail from evidence. Never report a
result you did not observe.

## Our standard: zero defects, fixed by a patch release

Smithers holds itself to a high bar. **We do not accept failures.** If you find
ANY issue, record it as a `fail` with exact evidence: a minor one, a
documentation inconsistency, an edge case, an ugly or confusing error message, a
broken link, or something entirely unrelated to this release all count. We ship a
patch release to fix it right away. Never downgrade, excuse, or rationalize a
finding as "pre-existing", "acceptable", "cosmetic", "expected", "minor", or "out
of scope".

The ONLY legitimate `skipped` is a genuine limitation of the test machine itself:
no coding agent installed, or no model credentials, so an agent task literally
cannot run. A product, CLI, doc, gateway, or behavior problem is never `skipped`
and never a pass-with-a-caveat. It is a `fail`.

## Version under test

The exact version is: {props.version}

Pin it on EVERY invocation — run commands as `bunx smithers-orchestrator@VERSION SUBCOMMAND`, substituting VERSION with the version above (do not use `@latest`, do not re-derive it).

## Setup — always work in a brand-new folder

1. Clear caches so you test the registry, not a stale copy: `bun pm cache rm`,
   then remove any `bunx-*smithers-orchestrator*` dir from `$TMPDIR` (macOS:
   `rm -rf $(echo $TMPDIR)bunx-*smithers-orchestrator*`; Linux:
   `rm -rf $TMPDIR/bunx-*smithers-orchestrator*`).
2. `SMOKE_DIR=$(mktemp -d)/smoke-features && mkdir -p "$SMOKE_DIR" && cd "$SMOKE_DIR"`.
   Never reuse a folder. Never work in the monorepo.

## Baseline — run every time

- `init` succeeds on a fresh folder. Confirm `.smithers/package.json`,
  `.smithers/workflows/hello.tsx`, `.smithers/prompts/hello.mdx` exist, and that
  `.smithers/.gitignore` ignores `pg/` and `migrated.json` (added in 0.25.1).
- `.smithers/node_modules/.bin/smithers` resolves to
  `smithers-orchestrator/src/bin/smithers.js` (use `readlink`).
- `workflow list` prints the seeded workflows (hello, implement, review, plan,
  ralph, debug, audit, ...) without crashing.
- After `init`, runs must be visible through the normal `ps`/`inspect` CLI and
  through a workspace Gateway with no client-side backend selection.

## The ranked feature checklist — verify each, oldest-fix-still-holds

For each, run the command and record `pass | fail | skipped` with the exact
command and the key output line. A changelog bullet that claims "X now works"
but X still fails is a smoke-test **failure**.

1. **`features.workspace-control-plane`** (HIGH) — After `init`, runs must be
   visible to the normal `ps`/`inspect`/`output` CLI with no `--backend` flag,
   and to Gateway `listRuns`/`getRun`. Treat SQLite/PGlite/Postgres as opaque
   storage behind the Gateway; never query a store to prove visibility.
2. **`features.migrate`** (0.25.x, HIGH) — `migrate --to pglite` copies runs
   forward with per-table progress and writes `migrated.json`; restart the
   workspace Gateway and prove the copied run through `listRuns`/`getRun`.
   `migrate --to sqlite` reverses it (new in 0.25.2) and the restarted Gateway
   still exposes the run. `migrate --to postgres` with no `--url` fails fast
   reporting the missing url requirement (exit 4), NOT a masked source-store
   error.
3. **`features.gateway-client-control`** (HIGH) — boot one workspace Gateway,
   discover its verified URL with `gateway status --format json`, and prove
   list/get/launch/cancel through Gateway RPC. Client code must not parse the
   runtime state file, assume port 7331, import a store opener, or use
   `--backend` on a read/control command.
4. **`features.gateway-root-routing`** (0.25.2, MEDIUM) — boot `gateway` on a
   free port in the background; `GET /` returns 302 (Location to the console) OR
   200 text/html (a landing page), never a bare `404` NOT_FOUND JSON. `GET
   /health` returns ok. Kill it.
5. **`features.interactive-tty-guard`** (0.25.0, MEDIUM) — piped (non-TTY)
   `up --interactive` exits 4 with `INTERACTIVE_REQUIRES_TTY`; `smithers tui` is
   now an unknown/removed command (folded into `--interactive`).
6. **`features.memory`** (0.25.0, LOW) — `memory set workflow:demo k '{"v":1}'`,
   then `memory get workflow:demo k` prints `{"v":1}`, `memory list` includes it,
   `memory rm workflow:demo k` deletes it, and a follow-up `get` reports the
   missing fact cleanly.
7. **`features.workflow-skills-nextsteps`** (0.25.2, LOW) — `workflow skills`
   prints next-step guidance about `.smithers/skills` being Smithers-owned and
   that Claude Code / Codex do not auto-scan it (link/copy into `.agents/skills`).
8. **`features.mcp-add-fallback`** (0.24.2, LOW) — `mcp add` against an agent
   whose own `mcp add` errors prints a copy-paste manual fallback command rather
   than dying silently.
9. **`features.docs-bundles`** (LOW) — `docs` prints a non-empty `llms.txt`
   index; `docs-full` prints thousands of lines (not truncated) and exits 0.
   Confirm the Codex-first GPT-5.6 role split is present and exact: Sol
   (`gpt-5.6-sol`) for planning/orchestration/review, Terra
   (`gpt-5.6-terra`) for validation and tool-heavy mid-tier work, and Luna
   (`gpt-5.6-luna`) for implementation/research/cheap work. Claude and Kimi
   may appear only as unavailable-Codex fallbacks; `gpt-5.5` must not be
   described as a default, and stale `claude-sonnet-4-6` /
   `gpt-5.2-codex` defaults must not appear.
10. **`features.paused-exit3`** (0.25.0, MEDIUM) — a seeded workflow with an
    approval gate (e.g. `review`) parks the run; `up`/`workflow run` exits 3 and
    prints a "Run is paused (exit 3 = awaiting a decision, not a failure)" header
    with approve/deny/why CTAs; `ps` shows `waiting-approval`, not failed. If no
    agent/key is available to drive it to the gate, mark `skipped` with the
    reason — do NOT mark `fail` for an agent-auth issue.

Then scan the embedded changelogs below for any other user-facing bullet (new
command, new flag, fixed crash) not covered above and add a finding for it.

## A failing agent is not a failing feature

Some checks need a working coding agent (claude/codex) on PATH and credentials.
If a command fails ONLY because no agent is configured or a provider is out of
credits, mark it `skipped` with that exact reason. Reserve `fail` for an actual
Smithers/CLI regression (a crash, a wrong exit code, a removed-but-documented
command, an unknown-flag rejection, a 404 where routing was promised).

## Changelogs (source of truth)

Versions in scope: {props.versions}

The full changelog text for each version follows. Every bullet under a
features/fixes heading is a candidate test case:

{props.changelogs}

## Output

Return JSON matching the schema below. `passed` is `true` only if every baseline
check AND every non-skipped feature check passed. `findings`: one entry per check
with a dotted `area` (e.g. `features.migrate`), a `status`, and `evidence` (exact
command + the proving output line). `reproSteps`: if `passed` is `false`,
pasteable shell commands to reproduce each failure from a clean machine.

Required output shape (an example of the JSON to return):

{props.schema}
