# Temporal E2E — Mode 2 tiers: Steps 3–7

> Reference file for the **temporal-e2e-validate** skill — Mode 2, Steps 3–7.
> **Run `mode-2-setup.md` first** (Steps 1–2: Temporal server + worker processes) — every tier here assumes the server and worker(s) are already up.
> The **Timeouts policy** and the **surface-results-immediately** rule in `SKILL.md` apply to every command here.
> Sibling reference files: `routing-battery.md` (Step 8 — v1 routing), `queue-options-battery.md` (Step 9 — v2 queue options / worker-runtime profiles).

## Contents

- **Step 3** — Sequential tests: Tier 1 (sequence), Tier 2 (hydration), Tier 2b (cross-process registry), Tier 3 (parallel), Tier 4 (image generation), Tier 5 (image flow)
- **Step 4** — Verify graph output
- **Step 5** — Concurrent isolation tests (concept / pipe / multi-concept)
- **Step 5b** — Tier 8: cross-worker usage emission
- **Step 5c** — Tier 9: object generation cross-process
- **Step 5d** — Tier 11: `make_extract_pages` two-activity cross-process
- **Step 5e** — Tier 12: deeply-nested controller stack (CV batch screening)
- **Step 5f** — Tiers 13–16: error propagation across the activity → workflow → submitter boundary (LLM, extract, image-gen, and batched child-workflow failures)
- **Step 6** — StoragePayloadCodec tests: Tier 6 (codec transparency), Tier 7 (large payload)
- **Step 7** — Final report (master results table)

---

### Step 3: Sequential tests — run one at a time, report after each

Do NOT clean previous results automatically — the user may want to compare runs.

**Tier 1 — Can the worker execute a simple pipe sequence?**

This sends a 2-step PipeSequence (step_one → step_two) to a worker that has never seen
these pipes. The worker must unpack the LibraryCrate, register the pipes, and execute
them in order. This is the most basic "does Temporal work at all" test.

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \
  --pipe native_text_sequence \
  --temporal --dry-run --mock-inputs --no-logo --graph
```

After this completes, tell the user: PASS/FAIL, output dir, graph file path.

**Tier 2 — Can the worker handle concepts it has never seen? (in-process hydration)**

This sends a pipeline that defines a custom concept (`Greeting` with `message` and
`language` fields) at runtime. `WfPipeRouter` must register this dynamic class
into the per-workflow registry before hydrating the WorkingMemory. If the hydration
order is wrong **on the router**, this fails with
`KajsonDecoderError: Class 'Greeting' not found`.

> Note: in dry-run with `pipelex run bundle` (no `delivery_assignment`), this tier
> only exercises hydration **on the router process**, not cross-process. It will
> pass even if the runner's registry is cold — see Tier 2b for the deterministic
> cross-process repro.

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds \
  --pipe dynamic_greeting_sequence \
  --temporal --dry-run --mock-inputs --no-logo --graph
```

After this completes, tell the user: PASS/FAIL, output dir, graph file path.

**Tier 2b — Cross-process registry regression (deterministic, dry-run).**

Same bundle as Tier 2, but submitted with a `DeliveryAssignment` attached. This
mirrors the cloud / `start_pipeline` + webhook path: `wf_pipe_run.py:79` then
schedules `act_deliver` on the runner with `pipe_output` carrying a hydrated
`WorkingMemory`. Temporal's data converter on the **runner process** must decode
the dynamic concept class — which is the exact path the `wf_pipe_router.py:71-80`
in-process propagation hack does NOT cover.

If the registry propagation across processes is broken, this fails every time with:

```
KajsonDecoderError: Class '<bundle>__<DynamicConcept>' not found
  in module 'builtins' or global registry
ApplicationError: Failed decoding arguments
  → temporalio/worker/_activity.py:566 (data_converter.decode_wrapper)
```

The error surfaces on the **runner** tmux session, raised by the activity worker
*before* the activity body runs.

```bash
timeout 120 .venv/bin/python .claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py
```

To reproduce against a different bundle:

```bash
timeout 120 .venv/bin/python .claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py \
  --bundle <path/to/bundle.mthds> --pipe <pipe_code>
```

After this completes, tell the user:
- PASS/FAIL.
- If FAIL: capture the runner log with
  `tmux capture-pane -t temporal-worker-runner -p -S -300 | tail -120`
  and quote the `KajsonDecoderError` and `ApplicationError: Failed decoding arguments`
  lines verbatim, plus the activity name (`act_deliver`).

**Tier 3 — Do parallel branches execute as separate child workflows?**

This sends a PipeParallel controller that fans out into two branches (tone analysis and
length analysis), each running as its own child workflow on the worker. The branches
execute concurrently and their results merge back. The `--graph` flag here is especially
interesting: it shows cross-worker execution with child workflow branches in the
ReactFlow visualization.

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds \
  --pipe temporal_parallel_sequence \
  --temporal --dry-run --mock-inputs --no-logo --graph
```

After this completes, tell the user: PASS/FAIL, output dir, graph file path.

**Tier 4 — Can the worker handle image generation pipelines?**

**Important — split workers REQUIRED for this tier.** Run Tier 4 only when the
two scoped workers from Step 2 (`mode-2-setup.md`) are alive (`temporal-worker-router` and
`temporal-worker-runner`). On a single full worker, image-gen routing accidents
are silently masked: every activity ends up on one process so a stray
`task_queue=` kwarg pointing at the wrong queue would still execute correctly.
Split workers are the only way to surface that class of bug as a hang or
wrong-pool execution.

**Important:** This tier only catches payload size bugs in **live mode**. In dry-run,
mock images are tiny and will pass even without activity-level storage. If the user
wants to validate image payload handling, they must run live.

This is a critical payload-size test. Image generation produces large binary data
(base64-encoded images) that must be stored at the activity level and passed as
storage URIs through Temporal — not inline in the workflow payload. If storage is
missing, this will either fail with a Temporal payload size error or trigger
`PayloadSizeWarning` in the worker logs.

The `generate_crazy_image` bundle runs a 2-step PipeSequence: an LLM imagines a
scene description, then PipeImgGen renders it as an image. This exercises the full
image generation path through Temporal, including the custom `ImagePrompt` concept
(which refines `Text`).

Dry-run:

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/pipes/pipelines/crazy_image_generation.mthds \
  --pipe generate_crazy_image \
  --temporal --dry-run --mock-inputs --no-logo --graph
```

Live (real image generation — required to catch payload size bugs):

```bash
timeout 600 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/pipes/pipelines/crazy_image_generation.mthds \
  --pipe generate_crazy_image \
  --temporal --no-logo --graph
```

After this completes, tell the user: PASS/FAIL, output dir, graph file path.
Also check worker logs for `PayloadSizeWarning`:

```bash
tmux capture-pane -t temporal-worker-router -p -S -50 | grep -i "payload\|warning\|size" || echo "No payload warnings (router)"
tmux capture-pane -t temporal-worker-runner -p -S -50 | grep -i "payload\|warning\|size" || echo "No payload warnings (runner)"
```

> For per-activity routing validation that proves image-gen lands on a
> dedicated worker pool (via `activity_queues` override), see **Step 8 —
> Routing validation battery** in `routing-battery.md`.

**Tier 5 — Can generated images flow between pipes in a sequence?**

**Important — split workers REQUIRED for this tier.** Same as Tier 4 — the
mis-routing of image-gen across worker pools is only observable when router and
runner run as separate processes. On a single full worker, the bug surfaces as
"works on my machine" but breaks in production.

**Important:** Like Tier 4, this only catches real payload bugs in **live mode**.

This is the "image out → image in" test. A PipeSequence first generates an image
via PipeImgGen, then passes that image as input to a PipeLLM (vision model) that
describes it. This exercises:
- Image generation storage at the activity level
- Image content serialization through Temporal's data converter
- Image content deserialization on the worker for the next pipe step
- Vision model receiving an image reference (URI) rather than inline base64

If activity-level storage is not implemented, this will fail because the raw image
bytes exceed Temporal's payload limit, or because the worker can't deserialize the
`ImageContent` object from the previous step.

Dry-run:

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/pipes/pipelines/test_image_out_in.mthds \
  --pipe image_out_in \
  --temporal --dry-run --mock-inputs --no-logo --graph
```

Live (real image generation + vision — required to catch payload size bugs):

```bash
timeout 600 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/pipes/pipelines/test_image_out_in.mthds \
  --pipe image_out_in \
  --temporal --no-logo --graph
```

After this completes, tell the user: PASS/FAIL, output dir, graph file path.
Also check for payload warnings on both workers:

```bash
tmux capture-pane -t temporal-worker-router -p -S -50 | grep -i "payload\|warning\|size" || echo "No payload warnings (router)"
tmux capture-pane -t temporal-worker-runner -p -S -50 | grep -i "payload\|warning\|size" || echo "No payload warnings (runner)"
```

> For per-activity routing validation across the image-gen + LLM-text
> activities exercised here, see **Step 8 — Routing validation battery**
> in `routing-battery.md`.

If any tier hangs for more than 30 seconds, check worker output. With the
recommended split workers (Step 2, `mode-2-setup.md`), capture both sessions; for the
alternative single full worker setup, capture `temporal-worker` instead:

```bash
# Split workers (default setup from Step 2, mode-2-setup.md)
tmux capture-pane -t temporal-worker-router -p -S -100
tmux capture-pane -t temporal-worker-runner -p -S -100

# Single full worker (alternative setup)
tmux capture-pane -t temporal-worker -p -S -100
```

### Step 4: Verify graph output

Check that the `--graph` flag produced ReactFlow HTML files:

```bash
ls results/*/reactflow.html 2>/dev/null
```

Tell the user where the files are and propose opening the most interesting one (the
PipeParallel graph shows cross-worker execution):

```bash
open results/temporal_parallel_sequence_output_01/reactflow.html
```

If no `reactflow.html` exists, GraphSpec assembly failed — check that tracing is enabled
in `pipelex.toml` (`[pipelex.tracing_config]` with `is_enabled = true`).

### Step 5: Concurrent isolation tests — can conflicting workflows coexist?

These are the most important safety tests. They submit two workflows simultaneously to
the same worker, where both workflows define identically-named concepts or pipes but
with different structures. If per-workflow isolation is broken, one workflow will use
the other's class definitions and either crash or silently produce wrong data.

Each pair runs with `--graph`. The graph output from backgrounded jobs goes to separate
`results/<pipe_code>_output_NN/` dirs.

**Test A — Conflicting concept names:**
Two workflows both define a concept called `Result`, but alpha's has `score` + `label`
while beta's has `value` + `confidence` + `is_valid`. If isolation fails, one workflow
will try to populate the wrong fields.

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/conflict_concept_alpha.mthds \
  --pipe alpha_pipeline \
  --temporal --dry-run --mock-inputs --no-logo --graph &
PID_ALPHA=$!

timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/conflict_concept_beta.mthds \
  --pipe beta_pipeline \
  --temporal --dry-run --mock-inputs --no-logo --graph &
PID_BETA=$!

wait $PID_ALPHA && echo "Alpha: PASS" || echo "Alpha: FAIL"
wait $PID_BETA && echo "Beta: PASS" || echo "Beta: FAIL"
```

After this, tell the user both results.

**Test B — Conflicting pipe names:**
Two workflows both define a pipe called `shared_step` but with different prompts.
Does each workflow execute its own version?

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/conflict_pipe_alpha.mthds \
  --pipe pipe_alpha_pipeline \
  --temporal --dry-run --mock-inputs --no-logo --graph &
PID_ALPHA=$!

timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/conflict_pipe_beta.mthds \
  --pipe pipe_beta_pipeline \
  --temporal --dry-run --mock-inputs --no-logo --graph &
PID_BETA=$!

wait $PID_ALPHA && echo "Alpha: PASS" || echo "Alpha: FAIL"
wait $PID_BETA && echo "Beta: PASS" || echo "Beta: FAIL"
```

After this, tell the user both results.

**Test C — Worst case: multiple conflicting concepts at high concurrency:**
Two workflows define both `Profile` AND `Summary` with incompatible field structures.
This is the hardest isolation test — if ContextVar scoping leaks between workflows,
this is where it shows up.

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/multi_concept_alpha.mthds \
  --pipe multi_alpha_pipeline \
  --temporal --dry-run --mock-inputs --no-logo --graph &
PID_ALPHA=$!

timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/multi_concept_beta.mthds \
  --pipe multi_beta_pipeline \
  --temporal --dry-run --mock-inputs --no-logo --graph &
PID_BETA=$!

wait $PID_ALPHA && echo "Alpha: PASS" || echo "Alpha: FAIL"
wait $PID_BETA && echo "Beta: PASS" || echo "Beta: FAIL"
```

After this, tell the user both results.

### Step 5b: Tier 8 — Cross-worker usage emission (writer_id fallback)

This validates that `UsageReportEvent`s emitted from inference activities running on
the runner process land in the same NDJSON partition as the router-side trace events,
stamped with a runner-process `writer_id` (`act_{pid}_{uuid8}`). It's the e2e
counterpart to the Phase 4 integration test
`tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py`.

**Important — dry-run does NOT exercise this path.** In dry-run mode, `PipeLLM`
(and friends) instantiate `ContentGeneratorDry()` directly inside the workflow
body — on the router process — and never dispatch `act_llm_gen_text` to the
runner. So a vanilla `pipelex run bundle --temporal --dry-run` against
router+runner workers will emit ALL `usage_report` events with
`writer_id="primary"`, never `act_*`. To deterministically observe runner-side
fallback in dry-run, run the Phase 4 integration test, which substitutes the
inference activity with a wrapper that synthesizes a real `LLMJob` server-side:

```bash
timeout 180 .venv/bin/pytest -x -v \
  tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py \
  -m temporal --temporal-server local --timeout=60 2>&1 | tail -60
```

Expected: both test cases pass —
`test_runner_usage_event_lands_in_same_ndjson_dir` (lands in same dir, with an
`__w_act_*` file) and `test_no_double_emit_in_split_worker_pool` (no
double-counting between fast path and fallback).

**To observe `act_*` writer files via the CLI**, run live mode (real LLM call —
costs money) so `act_llm_gen_text` is actually dispatched to the runner:

```bash
timeout 600 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \
  --pipe native_text_sequence \
  --temporal --no-logo --graph
```

After completion, list the trace files for the run:

```bash
RUN_ID=$(ls -t .pipelex/traces/ | head -1)
ls -la .pipelex/traces/$RUN_ID/
grep -l '"event_kind":"usage_report"' .pipelex/traces/$RUN_ID/*.ndjson
grep -hoE '"writer_id":"[^"]+"' .pipelex/traces/$RUN_ID/*.ndjson | sort -u
```

Expect:

- At least one `wf_*__w_act_{pid}_{uuid}.ndjson` file alongside the
  router-side `wf_*.ndjson` files.
- The runner-side file contains `usage_report` events with
  `writer_id` starting `act_`.
- Router-side files contain `pipe_start` / `pipe_end_success` events with
  `writer_id="primary"`.

If only `writer_id="primary"` appears, the inference activity ran in the
router process. Check `worker_config.activity_queues` (and the workflow's
own `task_queue`) so that `act_llm_gen_text` resolves to a queue the runner
listens on, and confirm both workers are running with the latest code
(restart them if they predate the Phase 2 runner-side fallback commit).

### Step 5c: Tier 9 — Object generation through Temporal cross-process

Validates that `act_llm_gen_object` and `act_llm_gen_object_list` survive a
cross-process round-trip. Every existing `library_crate/*.mthds` outputs `Text`,
so the structured-output activities never go through the real Temporal data
converter cross-process — Tier 9 closes that gap. The `make_object` /
`make_object_list` round-trip uses
`model_dump(mode="json", serialize_as_any=True)` and `model_validate(...)` on
the activity boundary; if either side regresses, nested fields silently drop
or fail to validate.

The `structured_output_sequence` bundle defines an `Invoice` concept with a
nested `Customer` concept and a `LineItem` list — at least one nested field
is required so the test exercises non-trivial JSON serialization.

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/structured_output_sequence.mthds \
  --pipe generate_invoice_single \
  --temporal --dry-run --mock-inputs --no-logo --graph
```

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/structured_output_sequence.mthds \
  --pipe generate_invoice_list \
  --temporal --dry-run --mock-inputs --no-logo --graph
```

After each completes, tell the user: PASS/FAIL, output dir, graph file path.
The pytest counterpart in `tests/integration/pipelex/temporal/tracing/test_split_worker_object_gen.py`
runs the same scenario through the in-process server with split workers and
asserts the round-trip preserves nested field values exactly.

### Step 5d: Tier 11 — `make_extract_pages` two-activity cross-process

Validates the most fragile post-collapse path: the only `ContentGeneratorProtocol`
method that dispatches more than one activity. `make_extract_pages` runs
`act_extract_gen_extract_pages` then conditionally `act_render_page_views`,
attaching each rendered page to the corresponding `PageContent.page_view`. This
is also the only method whose `activity_id` uniqueness mitigation
(`f"{base_id}-pages"` and `f"{base_id}-render-page-views"`) actually matters in
production.

The test fixture workflow `WfTestContentGeneratorPdfPageViews` (registered in
`TEMPORAL_TEST_WORKFLOWS`) exercises this flow. The pytest counterpart
substitutes both activities with canonical fixtures so it runs without Azure
Document Intelligence credentials or pypdfium2 — the goal is to pin the
activity_id contract, not to re-validate the OCR backend (already covered by
`content_generation/test_tprl_content_generator_pdf_page_views.py`):

```bash
timeout 180 .venv/bin/pytest -x -v \
  tests/integration/pipelex/temporal/tracing/test_split_worker_extract_pages.py \
  -m temporal --temporal-server local --timeout=60 2>&1 | tail -60
```

After this completes, tell the user: PASS/FAIL. The test asserts via
`WorkflowHandle.fetch_history()` that exactly two `ActivityTaskScheduled`
events appear with `activity_id` ending in `-pages` and `-render-page-views`.
For manual Temporal Web UI inspection, point `temporal-worker-router` /
`temporal-worker-runner` at the same dev server and inspect the workflow's
event timeline — the two scheduled activities should be visible with their
distinct activity_ids.

### Step 5e: Tier 12 — Deeply-nested controller stack (CV batch screening)

Validates the full pipelex-demos example 21 pipeline through Temporal: a
nested PipeSequence -> PipeSequence (job-offer prep) + PipeBatch (per-CV
fan-out) -> PipeSequence (extract + analyze + match) over PipeExtract +
PipeLLM operators. This is the canonical "real-world" e2e bundle for the
skill — controller composition that the other tiers only exercise piecewise.

The pytest counterpart lives at
`tests/integration/pipelex/temporal/library_crate/test_wf_cv_batch_screening.py`
(in-process server, dry mode). The direct-mode (no Temporal) e2e counterpart
lives at `tests/e2e/pipelex/cv_batch_screening/test_cv_batch_screening.py`.

Dry-run (CLI mocks inputs — no PDFs needed):

```bash
timeout 180 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/cv_batch_screening.mthds \
  --pipe batch_analyze_cvs_for_job_offer \
  --temporal --dry-run --mock-inputs --no-logo --graph
```

Live (real Azure Doc Intel extract + LLM analysis — uses `John-Doe-CV.pdf`
and `Job-Offer.pdf` from `tests/data/documents/`):

```bash
timeout 900 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/cv_batch_screening.mthds \
  --pipe batch_analyze_cvs_for_job_offer \
  --inputs tests/integration/pipelex/temporal/library_crate/cv_batch_screening_inputs.json \
  --temporal --no-logo --graph
```

After completion, tell the user: PASS/FAIL, output dir, graph file path. The
generated ReactFlow graph is unusually rich for this pipeline (nested
PipeSequence containers + PipeBatch fan-out edges + per-step extract/LLM
nodes) and is the recommended one to open at the end of the run:

```bash
open results/batch_analyze_cvs_for_job_offer_output_01/reactflow.html
```

If the live run errors with `Extract choice '...' was not found in the model
deck`, the deck is not loading Azure Doc Intel — fix the deck before falling
back to a different handle; user preference is Azure Doc Intel via the
Pipelex Gateway, period (do not substitute `mistral-ocr` / `deepseek-ocr`).

### Step 5f: Tiers 13–16 — Error propagation across the activity → workflow → submitter boundary

These four tiers validate the error-handling deliverable in a *true* 3-process
setup: a worker-side failure must travel back to the submitter carrying its
`ErrorReport` — the submitter must surface the *real classified inner error*,
not a bare "workflow failed" wrapper. The in-process pytest suite (Mode 1 Step 2b
in `SKILL.md`) asserts the structured `ErrorReport` fields directly; these tiers
prove the same survives real OS-process serialization. Tiers 13–15 fail a
*single* top-level activity, one per activity family (LLM, extract, image-gen);
Tier 16 fails inside a fanned-out child workflow.

**Submitter note.** The human CLI (`pipelex run bundle`) is the only CLI that
dispatches to Temporal — the agent CLI has no `--temporal`. It prints a flat
message, so the PASS check is at the message level. Precise field-level
assertions (`error_category` etc.) belong to Mode 1 Step 2b in `SKILL.md`.

**Inducing the failure — why `.env` must be edited.** The worker is a separate
process; you make its inference calls fail by giving it invalid credentials.
A shell-env prefix on the `tmux` command does NOT work: Pipelex loads `.env`
with `override=True` (`pipelex/system/environment.py`), so `.env` always wins
over the shell env. The reliable method is to temporarily append invalid keys
to `.env`, start the worker (it captures the tampered env at boot), then restore
`.env` immediately — the submitter and everything after run with a clean `.env`,
while the already-booted worker keeps the bad keys.

**Each tier owns the task queue exclusively.** The bad-credential worker must be
the *only* worker polling `temporal_task_queue` — else a healthy shared worker
picks up the activity and succeeds. Kill all workers first; restart the shared
ones (Step 2 in `mode-2-setup.md`) only if you continue to later tiers (Step 6 restarts its own).

**Start the bad-credential worker** — shared by Tiers 13–16, start it once:

```bash
# Back up .env, append invalid inference credentials, start the worker, restore .env
cp .env /tmp/dotenv-tier-err-bak
cat >> .env << 'EOF'

# --- temporal-e2e-validate Tier 13-16: invalid inference credentials (temporary) ---
OPENAI_API_KEY=invalid-key-tier-err
PIPELEX_GATEWAY_API_KEY=invalid-key-tier-err
PIPELEX_INFERENCE_API_KEY=invalid-key-tier-err
ANTHROPIC_API_KEY=invalid-key-tier-err
GOOGLE_API_KEY=invalid-key-tier-err
AZURE_API_KEY=invalid-key-tier-err
EOF
pkill -f "pipelex.temporal.worker_cli" 2>/dev/null; sleep 1
tmux kill-session -t temporal-worker-err 2>/dev/null
tmux new-session -d -c "$PWD" -s temporal-worker-err \
  '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed'
for attempt in $(seq 1 30); do
  if tmux capture-pane -t temporal-worker-err -p 2>/dev/null | grep -q "Temporal Worker started"; then break; fi
  if [ "$attempt" -eq 30 ]; then
    echo "TIMEOUT: temporal-worker-err did not start within 30s — last 50 lines:"
    tmux capture-pane -t temporal-worker-err -p -S -50
  fi
  sleep 1
done
cp /tmp/dotenv-tier-err-bak .env && echo ".env restored — the worker holds the invalid keys"
grep -q "tier-err" .env && echo "WARNING: .env still tampered — restore manually" || echo ".env clean"
```

**Tier 13 — LLM activity error propagation**

Submits `native_text_sequence` live. The first `act_llm_gen_text` call fails
provider-side with a 401 auth error — classified `CONFIGURATION` (non-retryable,
so no retry storm), converted to `TemporalError`, carried back as an
`ErrorReport`. The auth rejection happens before any tokens are billed — free.

```bash
timeout 600 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \
  --pipe native_text_sequence \
  --temporal --no-logo > /tmp/tier13.log 2>&1
echo "EXIT=$?"
tail -25 /tmp/tier13.log
```

Report to the user:

- **PASS** = non-zero exit (within ~30s — `CONFIGURATION` is non-retryable), and
  the final `Failed to execute pipeline '...': <cause>` line carries the *real
  inner error* — an authentication / `401` / `Invalid API Key` message. That
  proves the worker's `ErrorReport` crossed the boundary: without recovery the
  cause would be a bare workflow-failed wrapper with no inner detail. Confirm on
  the worker:
  `tmux capture-pane -t temporal-worker-err -p -S -200 | grep -iE "act_llm_gen_text|TemporalError|authentication|401"`.
- **FAIL** = hangs > 60s; exits 0 (a stray healthy worker stole the activity —
  check no other worker polls the queue); or the cause is only a generic
  workflow-failed wrapper with no inner provider error (see "Interpreting
  failures").

Note: the string `Failed to execute workflow WfPipeRun:` appears in a normal
`ERROR` log line — that is expected. The signal is the *inner cause* after it.

**Tier 14 — non-LLM (extract) activity error propagation**

The counterpart through a non-LLM activity, against the *same* bad-credential
worker. `act_extract_gen_extract_pages` fails on the worker; the `ErrorReport`
must cross the same way. Mirrors the LLM + non-LLM split that
`test_activity_error_boundary.py` covers in-process.

Create the inputs JSON if it does not exist:

```bash
cat > /tmp/pdf_extract_inputs.json << EOF
{
  "source_pdf": {
    "concept": "native.Document",
    "content": { "url": "$PWD/tests/data/documents/Job-Offer.pdf" }
  }
}
EOF
```

```bash
timeout 600 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/pdf_extract_page_views.mthds \
  --pipe pdf_extract_with_page_views \
  --inputs /tmp/pdf_extract_inputs.json \
  --temporal --no-logo > /tmp/tier14.log 2>&1
echo "EXIT=$?"
tail -25 /tmp/tier14.log
```

Report to the user:

- **PASS** = non-zero exit, and the `Failed to execute pipeline '...': <cause>`
  line carries the real extract auth error (`Extract service error ... 401 ...
  Invalid API Key`). Confirm on the worker:
  `tmux capture-pane -t temporal-worker-err -p -S -200 | grep -iE "act_extract_gen_extract_pages|TemporalError|authentication|401"`.
- **FAIL** = same modes as Tier 13.

**Tier 15 — image-generation activity error propagation**

The counterpart through the image-generation activity (`act_img_gen_images`),
against the *same* bad-credential worker. The error boundary
(`@convert_pipelex_errors`) is wired on every in-scope activity — Tier 13 proves
it for LLM, Tier 14 for extract, this tier closes it for image generation.

`generate_image` is a single `PipeImgGen` step with a static prompt, so
`act_img_gen_images` is the first (and only) inference activity dispatched — a
failure here is unambiguously the image-gen path, not a preceding LLM step. The
image model (`gpt-image-1-mini`) rejects the request on the tampered
OpenAI / Gateway key before any image is billed — free.

```bash
timeout 600 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/pipes/pipelines/test_image_out_in.mthds \
  --pipe generate_image \
  --temporal --no-logo > /tmp/tier15.log 2>&1
echo "EXIT=$?"
tail -25 /tmp/tier15.log
```

Report to the user:

- **PASS** = non-zero exit, and the `Failed to execute pipeline '...': <cause>`
  line carries the real image-gen auth error (`401` / `Invalid API Key` from the
  image model). Confirm on the worker:
  `tmux capture-pane -t temporal-worker-err -p -S -200 | grep -iE "act_img_gen_images|TemporalError|authentication|401"`.
- **FAIL** = same modes as Tier 13.

**Tier 16 — error propagation out of a fanned-out child workflow (PipeBatch)**

Tiers 13–15 fail a *single* top-level activity. This tier proves the report also
crosses when the failure happens *inside a fanned-out child workflow*: a
`PipeBatch` dispatches one child workflow per item, and each child's
`act_llm_gen_text` fails on the bad-credential worker. The first failing child's
`ErrorReport` must propagate up through the batch controller → `WfPipeRouter` →
submitter — not collapse into a generic "a batch item failed" wrapper.

Running the `PipeBatch` pipe (`batch_temporal_describe_topics`) directly — not
the wrapping `temporal_batch_sequence` — is deliberate: the sequence would fail
at its first LLM step (topic generation) before ever reaching the fan-out.
Feeding the topic list via `--inputs` skips topic generation so the batch
fan-out is the first inference dispatched.

Create the inputs JSON:

```bash
cat > /tmp/batch_topics_inputs.json << 'EOF'
{
  "topics": {
    "concept": "temporal_batch_test.Topic",
    "content": ["science", "history", "music"]
  }
}
EOF
```

```bash
timeout 600 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds \
  --pipe batch_temporal_describe_topics \
  --inputs /tmp/batch_topics_inputs.json \
  --temporal --no-logo > /tmp/tier16.log 2>&1
echo "EXIT=$?"
tail -25 /tmp/tier16.log
```

Report to the user:

- **PASS** = non-zero exit, and the `Failed to execute pipeline '...': <cause>`
  line carries the real per-branch auth error (`401` / `Invalid API Key`) —
  proving the child workflow's `ErrorReport` survived the fan-in, not a generic
  batch wrapper. Confirm on the worker:
  `tmux capture-pane -t temporal-worker-err -p -S -200 | grep -iE "act_llm_gen_text|temporal_describe_topic|TemporalError|401"`.
- **FAIL** = hangs > 60s (a retry storm — the per-branch failure should be
  non-retryable `CONFIGURATION`); exits 0 (a stray healthy worker stole a
  branch activity); or the cause is a generic batch/workflow-failed wrapper with
  no inner provider error (the child `ErrorReport` did not cross the fan-in —
  the exact gap this tier exists to catch).

**Teardown** (after Tiers 13–16):

```bash
tmux kill-session -t temporal-worker-err 2>/dev/null
# Safety net: ensure .env was restored, then drop the backup
if [ -f /tmp/dotenv-tier-err-bak ]; then
  grep -q "tier-err" .env && cp /tmp/dotenv-tier-err-bak .env && echo ".env restored from backup"
  rm /tmp/dotenv-tier-err-bak
fi
grep -q "tier-err" .env && echo "WARNING: .env still tampered" || echo ".env clean"
```

If you continue to Step 6 or later tiers, restart the shared worker(s) per Step 2 in `mode-2-setup.md`.

---

### Step 6: StoragePayloadCodec tests — does the codec work end-to-end?

These tests validate Phase 5: large payloads are transparently offloaded to external
storage by the `StoragePayloadCodec`, keeping Temporal's event history small. The worker
and submitter both use the codec — it's wired at the data converter level.

**Important:** The codec config is managed via `pipelex_temporary_override.toml` — a
config layer that loads after `pipelex_override.toml` and takes highest priority. This
file is ephemeral and gitignored. **NEVER modify `pipelex_override.toml`** — that is
the user's personal config.

**Step 6a: Create the temporary override to enable the codec**

```bash
cat > .pipelex/pipelex_temporary_override.toml << 'EOF'
# Temporary override for E2E codec testing — delete when done
[temporal.payload_codec_config]
is_enabled = true
EOF
echo "Temporary override created"
```

This takes precedence over whatever codec setting exists in `pipelex_override.toml`.

**Step 6b: Restart the worker** (so it picks up the new config)

```bash
tmux has-session -t temporal-worker 2>/dev/null && tmux kill-session -t temporal-worker
tmux new-session -d -c "$PWD" -s temporal-worker \
  '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed'
sleep 4
tmux capture-pane -t temporal-worker -p -S -30 | grep -i "payload codec\|Temporal Worker started"
```

Look for `Payload codec enabled` and `Temporal Worker started`.

**Tier 6 — Codec transparency: existing pipelines work unchanged**

Re-runs Tier 1 and Tier 2 with the codec on. If the codec is truly transparent, the
same pipelines produce the same results. The only difference: payloads above 1MB are
stored in `.pipelex/temporal-payload-store/` instead of inline in Temporal's event history.

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \
  --pipe native_text_sequence \
  --temporal --dry-run --mock-inputs --no-logo --graph
```

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds \
  --pipe dynamic_greeting_sequence \
  --temporal --dry-run --mock-inputs --no-logo --graph
```

After both complete, check that storage was used:

```bash
ls -la .pipelex/temporal-payload-store/ 2>/dev/null && echo "Storage files found" || echo "No storage files (payloads may be below threshold in dry-run)"
```

Tell the user: PASS/FAIL for each, and whether storage files were created. In dry-run
mode, payloads may be below the 1MB threshold, so no storage files is expected. In live
mode, larger payloads should trigger storage. Either way, the key assertion is that the
pipelines complete successfully.

**Tier 7 — Large payload stress test**

A 3-step pipeline that accumulates verbose text output across steps. Each step's
WorkingMemory carries the results of all previous steps, so the payload grows. In live
mode this produces realistic multi-KB to multi-MB payloads. In dry-run mode, mock
outputs are small but the codec path is still exercised.

```bash
timeout 120 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/large_payload_sequence.mthds \
  --pipe large_payload_sequence \
  --temporal --dry-run --mock-inputs --no-logo --graph
```

Live (real LLM calls — produces larger payloads, better stress test):

```bash
timeout 600 .venv/bin/pipelex run bundle \
  tests/integration/pipelex/temporal/library_crate/large_payload_sequence.mthds \
  --pipe large_payload_sequence \
  --temporal --no-logo --graph
```

After this completes, check storage and worker logs:

```bash
ls -la .pipelex/temporal-payload-store/ 2>/dev/null | head -20
tmux capture-pane -t temporal-worker -p -S -50 | grep -i "payload\|warning\|size\|codec" || echo "No payload warnings"
```

Tell the user: PASS/FAIL, output dir, graph file path, storage file count.

**Step 6c: Remove the temporary override**

```bash
.venv/bin/python -c "from pathlib import Path; Path('.pipelex/pipelex_temporary_override.toml').unlink(missing_ok=True)"
echo "Temporary override removed"
```

Optionally restart the worker to restore the base codec config, or leave it if done.

### Step 7: Final report

List all graph files and present a summary table with results. Tier 10a–10c are defined in `routing-battery.md` and Scenarios A–F in `queue-options-battery.md` — include those rows only if you ran those batteries:

```bash
ls results/*/reactflow.html
```

| Test | What it proved | Status | Graph | Payload warnings |
|------|---------------|--------|-------|-----------------|
| Tier 1: Sequence | Worker can unpack crate and run a pipe sequence | PASS/FAIL | path | — |
| Tier 2: Hydration | Worker handles dynamic concepts it has never seen | PASS/FAIL | path | — |
| Tier 2b: Cross-process registry | `act_deliver` decodes hydrated `pipe_output` on runner (forced via `delivery_assignment`) | PASS/FAIL | — | — |
| Tier 3: Parallel | Branches execute as concurrent child workflows | PASS/FAIL | path | — |
| Tier 4: ImgGen | Image generation pipeline works through Temporal | PASS/FAIL | path | yes/no |
| Tier 5: Image flow | Generated image flows as input to next pipe step | PASS/FAIL | path | yes/no |
| Tier 6: Codec transparency | Existing pipelines work unchanged with codec enabled | PASS/FAIL | path | — |
| Tier 7: Large payload | Multi-step pipeline with codec stress test | PASS/FAIL | path | — |
| Tier 8: Cross-worker usage | Runner-side `UsageReportEvent` lands in same NDJSON dir with `act_*` writer_id (live mode or integration test) | PASS/FAIL | — | — |
| Tier 9: Object gen cross-process | `act_llm_gen_object` / `act_llm_gen_object_list` survive the JSON round-trip with nested fields intact | PASS/FAIL | path | — |
| Tier 10a: Multi-activity routing | `activity_queues.default` routes both `act_llm_gen_text` and `act_img_gen_images` to their dedicated worker pools; default runner sees 0 hits for either | PASS/FAIL/SKIPPED | — | — |
| Tier 10b: Per-handle routing | `activity_queues.by_handle` overrides the activity default per model handle — two distinct handles in one workflow land on two distinct workers | PASS/FAIL/SKIPPED | path | — |
| Tier 10c: Two activities, one route | `act_extract_gen_extract_pages` + `act_render_page_views` (no routing key) both land on a shared dedicated queue via Azure Doc Intel through Pipelex Gateway | PASS/FAIL | — | — |
| Tier 11: Extract two-activity | `act_extract_gen_extract_pages` + `act_render_page_views` dispatched cross-process with distinct activity_ids | PASS/FAIL | — | — |
| Tier 12: CV batch screening | Deeply-nested controller stack (PipeSequence → PipeSequence + PipeBatch → PipeSequence) over PipeExtract + PipeLLM (pipelex-demos example 21) | PASS/FAIL | path | — |
| Tier 13: LLM error propagation | A worker-side LLM auth failure crosses activity→workflow→submitter; submitter shows the real error, not the generic `"Failed to execute workflow"` wrapper | PASS/FAIL | — | — |
| Tier 14: Extract error propagation | Same for a non-LLM (`act_extract_gen_extract_pages`) activity failure | PASS/FAIL | — | — |
| Tier 15: Image-gen error propagation | Same for the image-generation (`act_img_gen_images`) activity failure | PASS/FAIL | — | — |
| Tier 16: Batch child-workflow error propagation | A per-branch failure inside a `PipeBatch` fan-out child workflow crosses the fan-in carrying its `ErrorReport`, not a generic batch wrapper | PASS/FAIL | — | — |
| Scenario A: v2 multi-class routing | Specialized scopes + per-class runners cover LLM/img-gen/extract without cross-class leakage | PASS/FAIL/SKIPPED | — | — |
| Scenario B: per-queue timeout | `queue_options[X].start_to_close_timeout` overrides the worker_config baseline and flows into `ActivityTaskScheduled.start_to_close_timeout` | PASS/FAIL/SKIPPED | — | — |
| Scenario C: per-handle override | `handle_options[<handle>].start_to_close_timeout` wins over per-queue value for that one handle | PASS/FAIL/SKIPPED | — | — |
| Scenario D: queue rate limit | `queue_options[X].max_task_queue_activities_per_second` throttles dispatch — burst of N at rate R produces non-zero schedule_to_start latency on the tail | PASS/FAIL/SKIPPED | — | — |
| Scenario E: missing-worker timeout | A queue referenced by routing but polled by no worker produces a bounded `schedule_to_start` timeout, not a hang | PASS/FAIL/SKIPPED | — | — |
| Scenario F: CLI typo | `--task-queue typo_q` fails fast with `WorkerTaskQueueUnknownError` and a "Did you mean?" suggestion | PASS/FAIL | — | — |
| Concept isolation (alpha) | Conflicting `Result` concepts don't cross-contaminate | PASS/FAIL | path | — |
| Concept isolation (beta) | (same test, other side) | PASS/FAIL | path | — |
| Pipe isolation (alpha) | Conflicting `shared_step` pipes use correct version | PASS/FAIL | path | — |
| Pipe isolation (beta) | (same test, other side) | PASS/FAIL | path | — |
| Multi-concept (alpha) | Two overlapping concepts with incompatible structures | PASS/FAIL | path | — |
| Multi-concept (beta) | (same test, other side) | PASS/FAIL | path | — |

After reporting, propose opening the PipeParallel graph (most interesting cross-worker view):

```bash
open results/temporal_parallel_sequence_output_01/reactflow.html
```

If any concurrent tests fail, capture worker output for diagnosis (split workers
from Step 2 in `mode-2-setup.md` are the default — fall back to `temporal-worker`
only if you used the single full worker setup):

```bash
tmux capture-pane -t temporal-worker-router -p -S -200
tmux capture-pane -t temporal-worker-runner -p -S -200
```
