real-stack-e2e · plan · 2026-06-11

The whole UI, end to end,
with zero mocks. Today.

One durable Smithers workflow drives apps/smithers to a new Playwright suite where every backend is real: Plue in Docker gateway in the cwd real Gemini Flash now, Cerebras when ready, real Claude. After the six base tickets it enters a ralph quality loop: Claude Fable plans and reviews, Codex implements, and every iteration must pass the full real gate. It blocks on you only when it genuinely needs a human.

Preflight as of this morning

Docker daemon up claude CLI 2.1.173 on PATH (subscription auth) codex CLI 0.139.0 + ChatGPT auth present ../plue checkout; seeds a real admin token (sign-in needs no setup) GEMINI_API_KEY set → chat runs real Gemini Flash until Cerebras lands

Workflow file: .smithers/workflows/real-stack-e2e.tsx (graph-validated). Navigate with ← →.

definition of done

One command, green from a cold start

pnpm -C apps/smithers typecheck
&& pnpm -C apps/smithers test:unit
&& pnpm -C apps/smithers exec playwright test --config playwright.real.config.ts

Existing fixture suite stays byte-identical

playwright.config.ts + tests/e2e/** (39 specs, ports 5275-5292, fake Plue, fake upstream). The audit step fails any ticket that touches it. The real suite is additive.

New real suite no mocks anywhere

playwright.real.config.ts + tests/e2e-real/**. No page.route(), no fixture imports, no fallback data. Enforced by a deterministic grep audit on every iteration.

The six specs that define "working e2e"

spec	proves
stack.spec.ts	Vite proxies reach the real gateway (`/health` JSON) and real Plue (`/api/user` 401 JSON); the app shell renders.
signin.spec.ts	Token sign-in lands as the real seeded Plue user "Alice Dev" and survives reload.
chat.spec.ts	Chat streams a real Cerebras completion through the real Worker code.
gatewayRun.spec.ts	A gateway workflow run makes a real Claude call and finishes, watched live in the UI.
approval.spec.ts	A run pauses on a human approval; approving from the UI resumes it to finished.
full suite (T6)	All of the above in one run, plus typecheck + unit tests, twice in a row.
ralph loop	Then continuous improvement: each iteration adds missing e2e specs, missing unit tests, or quality fixes, and must keep the full gate green.

target architecture

Everything behind one origin, every leg real

Port map

port	what	rule
4000	Plue api (compose-fixed)	health gate: `GET /api/health`
7342	e2e gateway, local bun process	booted/reused by the Playwright webServer
7331	dev gateway	never clobbered, never killed
5375 / 5376	real app origin / real Worker leg	new, collision-free
5275-5292	fixture suite	reserved, untouched

leg 1 · smithers cloud

Plue: the real cloud, locally, in Docker

Sign-in needs zero setup

The seed inserts user alice ("Alice Dev", admin) and the literal access token smithers_deadbeef…beef with its sha-256 hash in access_tokens. The api validates Authorization: Bearer by hash lookup. The sign-in spec pastes that real token into the real token box.

Already e2e-friendly

The compose file ships SMITHERS_ENABLE_E2E_TEST_ROUTES=true (mint routes under /api/_test/) and pins SMITHERS_SERVER_ADDR=:4000. Boot contract is proven daily by scripts/dev-with-plue.sh: compose up -d postgres migrate seed repo-host api, then poll /api/health.

New helper: scripts/e2e-real/plue-up.sh wraps exactly that contract for the Playwright webServer (boots, health-waits, then sleeps; down/status for humans; idempotent).

leg 2 · orchestration

The gateway runs in the cwd, as itself

No container, no image build. The same entrypoint dev uses, on a dedicated port: PORT=7342 bun .smithers/gateway.ts, owned by a Playwright webServer entry and reused when already healthy.

Why this beats the docker draft: the earlier draft containerized the gateway, which forced a CLAUDE_CODE_OAUTH_TOKEN hand-off into the image. In the cwd the gateway spawns the host claude CLI, which is already logged in. One human gate deleted.
Auto-mount: gateway.ts registers every workflows/*.tsx at boot, so the two tiny probe workflows (one agent call; one approval gate) register by existing. A stale 7342 gateway gets restarted to pick them up; 7331 is never touched.
Shared smithers.db: the dev and e2e gateways read the same run store. Specs therefore assert on the runId they launched, never on list counts, and the real config runs workers: 1.

the machine that does the work

One durable workflow, six phases

why the agent cannot cheat

Per-ticket loop: four independent gates

the six tickets

Each ticket is a spec that passes against the real stack

ticket	builds	verify (must exit 0)
t1-real-stack-boot	`plue-up.sh`, the cwd-gateway webServer leg, `playwright.real.config.ts`, `stack.spec.ts`	playwright --config playwright.real.config.ts tests/e2e-real/stack.spec.ts
t2-real-signin	token sign-in with plue's seeded alice token; asserts "Alice Dev" from real postgres; reload-safe; live-plue assumption test	… tests/e2e-real/signin.spec.ts
t3-real-chat-llm	`worker.ts` leg (refuses boot without CEREBRAS_API_KEY) + chat spec streaming a real completion	… tests/e2e-real/chat.spec.ts
t4-real-gateway-run	`e2e-probe.tsx` workflow (one real Claude task), host credential probe script, launch-and-watch spec	probe-agent-cred.sh && … tests/e2e-real/gatewayRun.spec.ts
t5-real-approval	`e2e-approval-probe.tsx` + spec: pause on approval, approve from the UI, run resumes to finished	… tests/e2e-real/approval.spec.ts
t6-real-suite-green	stabilization: whole suite in one run, docs, everything committed, idempotent twice in a row	typecheck && test:unit && full real suite

Tickets are also written to .smithers/tickets/real-stack-e2e/*.md at run start, so you can read exactly what each agent was told.

after the basics are green

Ralph quality loop: keep raising the bar

Once the six base tickets pass and push, the workflow stops following a fixed script and starts hunting: each iteration, Claude Fable plans a batch of 1-3 high-value items, Codex implements it, the full gate (typecheck + unit + entire real e2e suite) must stay green, Fable reviews the diff, and the iteration pushes. Up to 12 iterations, or until the planner declares the remaining work low-value.

where you come in

Human gates: rare, explicit, blocking

already satisfied chat upstream key

GEMINI_API_KEY is set, so /api/chat runs real Gemini Flash through Gemini's OpenAI-compatible endpoint, and preflight proves it with one real completion. When you get Cerebras set up, drop CEREBRAS_API_KEY into apps/smithers/.env.e2e.local (chmod 600, gitignored); it takes precedence automatically, no code change.

only if a probe fails claude + codex auth

Preflight makes one real claude -p call on claude-fable-5 and one real codex exec call on gpt-5.5. Both CLIs are installed and authenticated, so these should pass. If codex fails: codex login, or answer skipCodex: true to fall back to Claude implementation. If claude fails: claude /login or claude setup-token.

How a gate reaches you

# the run pauses (smithers up exits with code 3 = waiting on a human, by design)
bunx smithers-orchestrator human inbox
bunx smithers-orchestrator human answer <requestId> --value '{"cerebrasApiKey":"csk-…","claudeOauthToken":null,"anthropicApiKey":null,"note":null}'
bunx smithers-orchestrator workflow run real-stack-e2e --resume <RUN_ID>   # if the runner exited

Mid-run asks: implementation agents are instructed to raise ask-human when blocked on a credential or an irreversible step. Guessing and faking are prohibited by the ticket ground rules and caught by review.
T5 is intentionally human-shaped: the approval spec approves from the UI itself, so that gate needs no operator.
Secret hygiene: the audit greps every diff for csk- / sk-ant- prefixes; values never enter tracked files or logs.

eyes open

Risks and how the plan absorbs them

risk	mitigation
Real LLM latency and nondeterminism flake the suite	Behavioral assertions only; generous bounded timeouts; `workers: 1`; T6 exists purely to stabilize, and root-causes flakes instead of adding retries.
e2e gateway shares `smithers.db` with the dev gateway	Specs assert on the runId they launched; separate port 7342; the refuse-to-clobber rule from dev-with-plue.sh carries over.
Plue cold boot takes 30-60s and can mask failures	Health gate is `/api/health` JSON, never "port is open"; 240s webServer budget; reuseExistingServer makes warm runs fast.
Vite proxy and gateway WS quirks (custom-UI iframes use a root WS path that vite owns for HMR)	Specs assert run progress via the RPC path the app actually uses (poll + `/v1/rpc` with ws:true). The iframe WS limitation is dev-known and out of scope for these six specs.
OAuth buttons cannot complete on localhost (cross-origin cookie jar)	Token sign-in is the tested path; it is the documented reliable local flow.
Other agents share this working tree and git index	Explicit pathspecs only, no `git add -A`; sequential tickets; push happens once, after the full gate.
A ticket exhausts its 5 iterations	The run fails loudly with full verify/audit/review output persisted. Resume re-enters the loop after you intervene; nothing pretends to be green.

Launch it, watch it, answer it

# launch (from the repo root)
bunx smithers-orchestrator workflow run real-stack-e2e

# watch
bunx smithers-orchestrator ps
bunx smithers-orchestrator inspect <RUN_ID> --watch
bunx smithers-orchestrator logs <RUN_ID> --follow
bunx smithers-orchestrator why <RUN_ID>          # "why is it paused?"

# when it blocks on you
bunx smithers-orchestrator human inbox
bunx smithers-orchestrator human answer <id> --value '{…}'

Today, roughly

After T6: the base suite pushes, then the ralph loop runs up to 12 plan→implement→gate→review→push iterations (roughly 45-90 min each; the full real suite runs every iteration). It self-terminates when the planner judges the remaining work low-value.
Your attention is needed rarely: preflight only if a probe fails (all probes currently green), plus any mid-run ask-human. Everything else is agent wall-clock; the run is durable across interruptions.
Output you get: a green one-command real suite on main, growing e2e + unit coverage from the ralph loop, and artifacts/real-stack-e2e/report.md with the evidence, port map, and how-to-run docs.

The whole UI, end to end,with zero mocks. Today.