# Pickle Rick pipeline bug report — 2026-05-14

**Reporter:** max@loanlight.com
**Session:** `~/.codex/pickle-rick/sessions/2026-05-14-5b341205`
**Pipeline:** pickle → anatomy-park → szechuan-sauce
**Outcome:** cancelled mid-pickle phase after a parallel Claude session (in another terminal) issued conflicting git commands against the same checkout
**Severity:** S1 — data-loss / branch-corruption potential

---

## Validation Outcome — 2026-05-14 PM (3-agent parallel audit + operator correction)

**Operator correction 2026-05-14 PM (gregory):** reporter ran `pickle-rick-claude` (not the codex-distributed Pickle as my earlier framing assumed). The `~/.codex/pickle-rick/sessions/...` data path is configuration drift, not a different codebase. **All 5 bugs are filed as STILL-VALID against pickle-rick-claude**, even where the audit could not locate the named surfaces at current HEAD — that gap is root-cause discovery work owned by the fix tracks below, not a dismissal.

| Bug | Surface status at HEAD | Filing |
|---|---|---|
| 1 — `rg` in verification commands | Named surface (`refine-prd.js`, `verification` field in refinement manifest schema) not present at HEAD. AC phase gate (`extension/src/services/ac-phase-gate.ts:127-145`) DOES run operator-authored commands with `shell:true` — likely surface for the same bug class if commands contain `rg`. | **Filed as Master Plan Finding #39 (R-PVTA-1..6, P2)** |
| 2 — zsh `[bracket]` glob expansion | Node's `shell:true` on POSIX routes through `/bin/sh` (not zsh), but `/bin/sh` on macOS without `nullglob` still expands `[id]` and may error on no-match. Workers spawned through codex CLI (`codex exec`) may also route through user `$SHELL`. | **Filed as Master Plan Finding #40 (R-VSGE-1..5, P2)** |
| 3 — concurrent git access (S1) | All 5 sub-claims confirmed at HEAD (worktree isolation declared but unused, no HEAD pinning, prompts actively destructive, no lock cleanup, no concurrent-access probe). | **Filed as Master Plan Finding #37 (R-PIWG-1..8, P1)** |
| 4 — manifest blocked-state leak | `'Blocked'` TicketStatus value not in `transaction-ticket-ops.ts` at HEAD (only `'Todo' \| 'In Progress' \| 'Done' \| 'Skipped'`). Reporter clearly saw `done=0 blocked=1 skipped=0` runner output — either an earlier version had `'Blocked'`, or the symptom is a `'Skipped'` ticket being misrendered, or there's a separate runner-internal blocked-set distinct from `TicketStatus`. | **Filed as Master Plan Finding #41 (R-RMBS-1..6, P2)** |
| 5 — `--resume` cwd rigidity | Confirmed at HEAD: `setup.ts:595-606` `validateResumeCompatibility` hard-rejects cross-cwd resume instead of `process.chdir`-ing into the recorded `working_dir`. | **Filed as Master Plan Finding #38 (R-PRCR-1..4, P3)** |

Validation methodology: 3 parallel `general-purpose` agents (per-bug-cluster), source greps for named surfaces, and `prds/MASTER_PLAN.md` cross-check for sister filings. Where the named surface couldn't be located at HEAD, the fix track's R-*-1 ticket is **"locate the current surface in this codebase"** — bug class is preserved, attribution is deferred to the implementor who'll grep with a wider lens.

---

## Context

Ran `pickle-pipeline.js --prd ./pre-pr-review-prd.md` against a 16-ticket PRD generated by the `pre-pr-review` skill on a feature branch (`feat/audit-ui-redesign`, repo `/Users/maxklein/local Code/loanlight-api`). The pipeline failed three separate ways before being cancelled. The third failure mode is the dangerous one and the primary reason for this report.

## Bug 1 — Refinement synthesis emits verification commands assuming `rg` exists

**What happened:** Refinement synthesis produced verification arrays containing `rg -n "dtiRatio" packages/app/src packages/api/src` (ref-01) and `rg -n "auditRunId|creditRunId|staff-only|partner" docs/adr/credit-overview-linkage.md` (ref-08). My system has no `rg` (ripgrep). The pickle runner exited with `verification-contract-failed: command not found: rg` on the first ticket, halting the whole pipeline.

**Root cause:** Synthesis (`refine-prd.js` or its synthesis step) generates verification commands from a model prompt that doesn't enumerate available tools on the host. The model defaults to `rg` because it's common in modern codebases.

**Recommended fixes (in priority order):**
1. Pre-flight tool availability check at pipeline launch: probe `command -v rg`, `command -v fd`, `command -v jq`, etc., and pass the available set into the synthesis prompt as a constraint.
2. Synthesis prompt addition: *"Use only `grep`, `find`, `sed`, `awk`, `cat`, `head`, `tail`, and `test` for shell verification commands. Avoid `rg`, `fd`, `bat`, `jq`, or any tool that may not be installed."*
3. At runtime, transparently rewrite `rg <pattern> <paths>` → `grep -rn <pattern> <paths>` if `rg` is not on PATH.

## Bug 2 — Verification commands fail under zsh due to glob expansion of Next.js dynamic-route brackets

**What happened:** ref-02's verification ran `pnpm --filter @loanlight/app test -- src/app/results/audits/[auditId]/_components/__tests__/ResultsOverview.test.tsx`. zsh interprets `[auditId]` as a character-class glob and, with `nomatch` set (default in many zsh configs), errors out: `zsh:1: no matches found: src/app/results/audits/[auditId]/...`. The path exists but never gets to pnpm.

**Root cause:** `pipeline-runner` (or wherever `child_process.exec`/`spawn` is invoked) launches the verification commands through zsh without quoting path arguments. Next.js / SvelteKit / Remix all use `[param]` and `(group)` directories which collide with zsh globbing.

**Recommended fixes:**
1. Run verification commands via `bash -c '<cmd>'` instead of the user's `$SHELL`. bash defaults to `nullglob` off and doesn't error on unmatched globs.
2. If sticking with the user's shell, prepend `setopt no_nomatch` (zsh) / `shopt -s nullglob` (bash) before the verification command.
3. At synthesis time, auto-quote any token in `verification` that contains `[`, `]`, `(`, `)`, `{`, `}`, `*`, `?`, or `~`. Single-quote it so the shell passes it through verbatim.
4. Synthesis prompt addition: *"Always single-quote path arguments that may contain shell-special characters (e.g. Next.js dynamic routes `[id]` or route groups `(group)`). Example: `pnpm test -- 'src/app/(detail)/loan-audits/[runId]/page.tsx'`."*

## Bug 3 — No isolation against concurrent git operations on the same checkout (CRITICAL)

**What happened:** While ticket ref-04 was in progress, a parallel Claude Code session (running in a different terminal window against the same repo) executed `git stash → git checkout main → git pull → git checkout -b credit-rule-addition` per a user request. The pickle worker had no idea, and:

1. Lost the in-progress ref-04 working-tree state to the parallel session's `git stash` (auto-stash `stash@{1}` "On feat/audit-ui-redesign: WIP: audit-ui-redesign changes" — 13 files of legitimate ref-03/ref-04 changes).
2. The repo's HEAD switched from `feat/audit-ui-redesign` to `main` underneath the running pipeline.
3. The parallel session was interrupted before it could run `git pull` or `git checkout -b credit-rule-addition`, leaving the repo on stale local `main` (`dd0c8685`, behind `origin/main` `116708df`) with `.git/index.lock` orphaned at 16:01:25 and ~299 modified files in the working tree (likely the parallel session's transient state).
4. Pickle workers continued executing against the corrupted state until the user manually noticed and cancelled.

User's reflog (verbatim):
```
fdb0c4ab HEAD@{0}: checkout: moving from main to feat/audit-ui-redesign  ← user-driven recovery
dd0c8685 HEAD@{2}: checkout: moving from feat/audit-ui-redesign to main  ← parallel Claude session
```

Branches confirmed: `credit-rule-addition` does NOT exist (parallel session never reached the create step). `main` is at stale `dd0c8685`; `origin/main` is at `116708df`.

This is the most dangerous failure mode I encountered. Had the user not cancelled, pickle would have kept "implementing" tickets against an entirely wrong tree, then auto-committed against `main`. The original report misattributed this to a codex worker — the actual culprit was unsynchronized concurrent access. The mitigation surface is the same regardless of who ran the offending git command.

**Root cause:** Pickle pipelines have no exclusive lock on the working directory they were launched against. A user can spawn another shell, IDE, or AI session that issues conflicting git commands at any time, and the pickle runner has no way to detect or prevent it.

**Recommended fixes (highest priority):**

1. **Execute every pipeline in a git worktree dedicated to its session.** Pickle should `git worktree add ../<repo>-pickle-<sha> <branch>` at session bootstrap, set the worker's `cwd` to that worktree, and never touch the user's primary checkout. This makes the entire bug class physically impossible — concurrent commands in the user's main checkout can't affect the pipeline's worktree, and vice versa. (This repo already uses worktrees as a pattern — `loanlight-api`, `loanlight-api-bank-stmt`, `loanlight-api-main` — so the model is familiar to users.)

2. **Pre/post-tool HEAD pinning in the runner:** at session bootstrap, capture `git symbolic-ref HEAD` and `git rev-parse HEAD`. Before each ticket and after each worker tool call, re-check both. If `symbolic-ref` changed (branch switched) or `rev-parse` jumped without a recorded commit, abort the pipeline immediately with a clear error: *"Working tree was modified by an external process. Pipeline cannot continue safely. Repository is at <branch>@<sha>; expected <branch>@<sha>."*

3. **Worker prompt guard (defense in depth even if #1 lands):** *"Do NOT run `git checkout`, `git switch`, `git reset`, `git push`, `git pull`, `git stash`, `git rebase`, or modify `.git/`. You are pinned to the current branch. If you need to inspect another branch, use `git show <ref>:<path>` or `git log <ref>` only. The only allowed mutating git commands are `git add` (paths inside `output_artifacts`) and `git commit`."*

4. **Stale lock cleanup on cancel:** `cancel.js` should detect `.git/index.lock` older than the runner's last activity and `rm -f` it (after confirming no live `git` process holds it), so users don't have to debug stale locks.

5. **Detect concurrent-access risk at launch:** at `pickle-pipeline.js` start, check `lsof <repo>/.git` (or equivalent fcntl probe) and warn if another git process is currently active. Print the worktree-isolation recommendation as a hint.

## Bug 4 — Manifest edits don't fully reset blocked-ticket runner state

**What happened:** After patching ref-01's verification (rg→grep) and resetting `manifest.tickets[0].status` from `Blocked` to `Pending`, the resumed pipeline reported `Runnable Tickets: 9 (done=0 blocked=1 skipped=0)` and skipped ref-01 entirely, jumping to ref-02. Same thing recurred after I patched and reset ref-02 — the runner skipped both, started at ref-03.

**Root cause:** The runner's "blocked" set is computed from somewhere other than `refinement_manifest.json[].status`. Possibly it's persisted in `pipeline-runner.log` parsing or in an unwritten in-memory state, or it tracks "tickets that have ever failed in this session."

**Recommended fixes:**
1. Make `refinement_manifest.json[].status` the single source of truth for ticket runnability.
2. Document the resume contract: which fields can a user safely edit between runs to influence what gets retried? Today users have to grep through the runner code to figure this out.
3. Add a `--retry <ticket-id>` flag to `pickle-pipeline.js --resume` that explicitly forces the named ticket back into the runnable queue.

## Bug 5 — `--resume` rejects launches from a different `cwd` than the original

**What happened:** Tried `node /Users/maxklein/.codex/pickle-rick/bin/pickle-pipeline.js --resume <session-dir>` from my home directory; got `Cannot change pipeline target on resume.` The `pipeline.json` already records `target` and `working_dir`, so the resume should be runnable from anywhere.

**Recommended fix:** Honor the recorded `target` from `pipeline.json` instead of inferring from `process.cwd()`. If they differ, `cd` into the recorded target before launching the runner. (Or print a clear error: `error: pipeline target is X, current cwd is Y; cd X first or run with --target X`.)

## Suggested issue tracker filing

These are 5 distinct issues — recommend filing as separate tickets with these labels:
- Bug 3 → `S1 / safety / git-isolation` — fix first, blocks any wider rollout
- Bugs 1, 2, 4 → `S2 / pipeline-correctness` — runtime quality of life
- Bug 5 → `S3 / cli-ergonomics` — small papercut

## Forensics retained

- Pickle session: `~/.codex/pickle-rick/sessions/2026-05-14-5b341205/` (kept; includes runner log, refined manifest, all artifacts)
- User reflog: shows the unauthorized `git checkout` (visible via `git reflog` in the affected repo)
- Auto-stash from worker: `stash@{1}` "On feat/audit-ui-redesign: WIP: audit-ui-redesign changes" — 13 files, useful for verifying what ref-03/ref-04 actually produced before the corruption
