---
description: "Automatic pipeline generation for unrouted tasks via /do."
user-invocable: false
allowed-tools:
  - Read
  - Write
  - Bash
  - Grep
  - Glob
  - Agent
  - Edit
context: fork
routing:
  triggers:
    - auto-pipeline
    - ephemeral pipeline
    - no route found
    - unrouted task
  pairs_with:
    - chain-composer
    - pipeline-scaffolder
    - domain-research
    - pipeline-test-runner
  complexity: Complex
  category: meta
---

# Auto-Pipeline

## Overview

This pipeline operates as the automatic fallback for `/do` when no existing route matches a non-trivial request. It classifies the task type, selects and adapts a canonical chain, executes it with phase gates, and optionally crystallizes the pattern into a permanent pipeline.

**Key principle**: ALWAYS check the pipeline catalog first. If an existing pipeline covers 70%+ of the request, route to it instead of creating a new one. This prevents duplicate pipeline fragmentation and maintenance burden.

---

## Instructions

### Phase 0: DEDUP CHECK

**Goal**: Ensure we're not duplicating an existing pipeline.

**Why this matters**: ALWAYS check the pipeline catalog first. If an existing pipeline covers 70%+ of the request, route to it instead. Duplicate pipelines fragment routing, create maintenance burden, and confuse discovery. The dedup gate is a HARD BLOCK — apply it even when the request seems "slightly different."

**Step 1**: Run task type classification:
```bash
python3 ~/.claude/scripts/task-type-classifier.py --request "{user_request}" --check-catalog ~/.claude/skills/auto-pipeline/references/pipeline-catalog.json --json
```

**Step 2**: If the classifier returns `existing_pipeline`, STOP. Route to that pipeline instead. Display:
```
===================================================================
 ROUTING: Existing pipeline found
===================================================================
 Request matches: {pipeline_name} (coverage: {N}%)
 Routing to existing pipeline instead of creating new.
===================================================================
```

**Step 3**: If no existing pipeline matches, capture the classification result:
- `task_type`: one of 8 canonical types
- `chain`: the canonical chain for that type
- `steps`: step count
- `keywords_matched`: which keywords triggered this classification

**Gate**: Classification complete. No existing pipeline matches. Proceed to Phase 1.

### Phase 1: CHAIN SELECTION

**Goal**: Select the best canonical chain variant and extend to 8-12 steps.

**Why 8-12 steps?** Fewer than 8 steps means under-utilizing phase gates, which reduces verification points and quality. Each additional step adds a gate — a mandatory quality checkpoint. A 6-step chain that could be 10 is leaving opportunities for validation on the table. Extend all chains to this range.

**Why sequential?** Every step must complete before the next begins. No skipping, no parallel steps unless the step explicitly supports parallelism (RESEARCH, REVIEW). Phase gates enforce ordering and prevent state leakage between phases.

**Step 1**: Read `skills/workflow/references/chain-composer/references/canonical-chains.md` for the full canonical chain and its variants.

**Step 2**: Select the best variant based on request analysis:
- Does the request involve unstructured input? → add EXTRACT
- Does the request need multi-source research? → add COMPILE or use GATHER variant
- Does the request need audience targeting? → add GROUND
- Does the request need iterative improvement? → add REFINE
- Does the request have verifiable syntax? → add LINT
- Does the request need spec conformance? → add CONFORM
- Does the request need post-execution observation? → add MONITOR
- Does the request involve state changes? → add CHARACTERIZE before, COMPARE after

**Step 3**: Verify chain has 8-12 steps. If fewer than 8:
- Add ASSESS before any generation/execution step
- Add COMPILE between research and generation
- Add REFINE after validation
- Add SYNTHESIZE before reporting

**Step 4**: Apply operator profile gates. Chain behavior shifts based on execution context:
- Personal: remove APPROVE, PROMPT; reduce GUARD to branch-check only
- Work: add CONFORM after GENERATE; full GUARD
- CI: skip interaction steps; add NOTIFY
- Production: add SIMULATE, SNAPSHOT, APPROVE (mandatory)

**Gate**: Chain has 8-12 steps. All type compatibility rules satisfied. Proceed to Phase 2.

### Phase 2: CONTEXT CHECK (Toolkit Repo Detection)

**Goal**: Determine whether to crystallize immediately or run ephemeral.

**Toolkit repo rule**: If running in this repo (detected by `skills/workflow/references/auto-pipeline.md` existing in CWD), crystallize on first encounter. This repo IS the pipeline system — every pattern we extract becomes part of the toolkit. Capture the pattern immediately on the first run.

**Outside toolkit repo rule**: Wait for 3+ ephemeral executions in the same domain before crystallizing. This ensures the pattern is stable and not a one-off.

**Step 1**: Check if `skills/workflow/references/auto-pipeline.md` exists in CWD (indicates toolkit repo).

**Step 2**: If toolkit repo:
- Check learning.db for any prior ephemeral runs in this domain (informational only — crystallize regardless)
- Proceed to Phase 3 (CRYSTALLIZE) instead of Phase 4 (EPHEMERAL EXECUTE)

**Step 3**: If NOT toolkit repo:
- Query learning.db: `python3 ~/.claude/scripts/learning-db.py search "ephemeral {task_type}" --json`
- Count matching entries for this domain
- If 3+ found: proceed to Phase 3 (CRYSTALLIZE)
- If <3 found: proceed to Phase 4 (EPHEMERAL EXECUTE)

**Gate**: Execution path determined (crystallize vs ephemeral). Proceed to appropriate phase.

### Phase 3: CRYSTALLIZE (Create Permanent Pipeline)

**Goal**: Create a permanent pipeline from the adapted chain and wire it into routing.

**Why parallel research in CRYSTALLIZE?** Rule 12 is validated by A/B testing — parallel research agents produce 1.40-point quality gap over sequential grep-based research. Dispatch 3-4 parallel agents instead of running searches sequentially. Sequential research is banned in RESEARCH steps.

**Why 10 steps?** This phase itself is a full pipeline: DETECT → GATHER → RESEARCH (parallel) → COMPOSE → VALIDATE → SCAFFOLD → INTEGRATE → VERIFY → REGISTER → EXECUTE. Every step gates the next. If any step fails, we roll back to ephemeral for the current request and record the failure for investigation.

**Step 1 — DETECT**: Confirm crystallization threshold is met (toolkit repo = always; other = 3+ prior runs).

**Step 2 — GATHER**: If prior ephemeral runs exist, collect their chain descriptions and outcomes from learning.db.

**Step 3 — RESEARCH**: Use accumulated evidence to inform pipeline design. Dispatch 3 parallel research agents (never sequential — Rule 12):
- Agent 1: Analyze what steps worked in prior ephemeral runs
- Agent 2: Find similar existing pipelines to learn from
- Agent 3: Identify domain-specific references/scripts needed

**Step 4 — COMPOSE**: Build the final chain:
- Start from the adapted canonical chain (from Phase 1)
- Incorporate learnings from prior ephemeral runs (if any)
- Ensure 8-12 steps
- Validate type compatibility

**Step 5 — VALIDATE**: Run `python3 ~/.claude/scripts/artifact-utils.py validate-chain` against the composed chain. If validation fails, fall back to unmodified canonical chain and log the adaptation failure.

**Step 6 — SCAFFOLD**: Create the pipeline skill:
- Create `skills/workflow/references/{pipeline-name}.md` with full operator context
- Create `skills/workflow/references/{pipeline-name}/references/` if domain references are needed
- Follow the existing pipeline SKILL.md format (frontmatter + operator context + phases)

**Step 7 — INTEGRATE**: Wire into routing:
- Run `python3 ~/.claude/scripts/generate-skill-index.py` to update INDEX.json
- The routing-table-updater skill handles /do routing table updates

**Step 8 — VERIFY**: Confirm the new pipeline is discoverable:
- Check `skills/INDEX.json` contains the new pipeline
- Verify `skills/workflow/references/{name}.md` exists and parses

**Step 9 — REGISTER**: Record in learning.db:
- Mark all contributing ephemeral learnings as graduated
- Record the crystallization event

**Step 10 — EXECUTE**: Route the original request through the newly created permanent pipeline.

**Gate**: Pipeline created, integrated, verified. Original request executing through permanent pipeline. If any earlier step fails, fall back to ephemeral execution for this request and record the failure.

### Phase 4: EPHEMERAL EXECUTE

**Goal**: Execute the adapted chain inline with phase gates, without persistence.

**Why save intermediate artifacts?** Context is ephemeral; files are not. Save intermediate output at each phase to session-local files so work survives context compression and session boundaries.

**Step 1**: Display the ephemeral pipeline banner:
```
===================================================================
 ROUTING: [EPHEMERAL PIPELINE] {task_description}
===================================================================

 Task type: {task_type}
 Chain ({N} steps): {step1} → {step2} → ... → {stepN}
 Profile: {profile} ({profile_description})

 Note: No permanent pipeline exists for this domain.
       Running ephemeral chain with phase gates.

===================================================================
```

**Step 2**: Execute each step in sequence with gating:
- For each step in the chain:
  1. Announce: `[Phase {N}/{total}: {STEP_NAME}]`
  2. Execute the step's action (research, compile, generate, validate, etc.)
  3. Save output to session-local file: `/tmp/ephemeral-pipeline/{step_name}.md`
  4. Verify gate condition is met (step must complete, output must exist)
  5. Proceed to next step only after gate passes

**Step 3**: For RESEARCH steps (Rule 12 mandatory): dispatch 3-4 parallel agents. Never run grep/search commands sequentially — that is the banned pattern. Instead:
- Agent 1: Code/content analysis
- Agent 2: Usage patterns / ecosystem context
- Agent 3: Examples and reference material
- Agent 4 (optional): External documentation / API references

Collect all results in parallel, then synthesize.

**Step 4**: For REVIEW steps: dispatch 3+ parallel reviewers with different lenses (contrarian, skeptical, user advocate, etc.).

**Step 5**: For REFINE steps: iterate up to 3 times until validation passes. After 3 iterations, proceed with best effort output and log what remains unresolved.

**Step 6**: On completion, record learning for crystallization tracking:
```bash
python3 ~/.claude/scripts/learning-db.py learn --skill auto-pipeline "ephemeral {task_type} for {domain}: {chain_description}"
```

This enables the 3+ run threshold for crystallization outside the toolkit repo.

**Gate**: All steps executed. Output delivered. Learning recorded.

---

## Error Handling

### Error: "Classification returns no matches"
**Cause**: Request doesn't match any task type keywords.
**Solution**: Default to `analysis` type (broadest applicability). Log a learning about the unclassifiable request.

### Error: "Dedup gate blocks but user insists"
**Cause**: Existing pipeline matches 70%+ but user says it doesn't serve their needs.
**Solution**: User can say "create new pipeline for X, existing one doesn't cover Y." The explicit override bypasses dedup.

### Error: "Chain validation fails"
**Cause**: Type compatibility error in adapted chain.
**Solution**: Fall back to the unmodified canonical chain for that task type. Log the adaptation that failed.

### Error: "Crystallization fails mid-scaffold"
**Cause**: Pipeline creation blocked by creation gate or ADR enforcement.
**Solution**: Fall back to ephemeral execution for the current request. Record the failure for investigation.

---

## References

- [Canonical Chains](../chain-composer/references/canonical-chains.md) - 8 canonical chain templates with variants
- [Step Menu](../pipeline-scaffolder/references/step-menu.md) - 23 step families, 150+ steps, type compatibility matrix
- [Pipeline Catalog](references/pipeline-catalog.md) - Auto-generated catalog of all existing pipelines (dedup reference)
