---
number: 9987
title: 'docs: add ''The Dream Pipeline & Golden Path'' guide (#9985)'
author: tobiu
state: MERGED
createdAt: '2026-04-13T18:56:54Z'
updatedAt: '2026-04-13T19:03:49Z'
closedAt: '2026-04-13T19:03:43Z'
mergedAt: '2026-04-13T19:03:43Z'
head: agent/9985-dream-pipeline-guide
base: dev
url: 'https://github.com/neomjs/neo/pull/9987'
---
## Summary

Adds a comprehensive guide to the Dream Pipeline — the offline forecasting engine that digests raw session memories into structured graph intelligence and synthesizes a **Golden Path** (mathematically ranked roadmap of what to work on next).

## What This Covers

### The 6-Phase REM Pipeline
- **Phase 0:** Workspace Ingestion (filesystem → graph sync)
- **Phase 1:** Tri-Vector Extraction (LLM-based semantic graph, feature namespace, summary)
- **Phase 2:** Topological Conflict Detection (obsolete/superseded issue alerts)
- **Phase 3:** Deterministic Capability Gap Inference (`DOC_GAP`, `TEST_GAP`, `GUIDE_GAP`)
- **Phase 4:** Garbage Collection (Hebbian decay, Vector Apoptosis)
- **Phase 5:** Golden Path Synthesis (Hybrid GraphRAG scoring)

### The Scoring Algorithm
- **Pillar 1:** Semantic distance from ChromaDB frontier vector
- **Pillar 2:** Structural weight from SQLite edge topology
- **Priority formula:** `(semantic × 2.0) + (structural × 1.0)`
- Multipliers: bug boost (+1.0), community boost (+0.5), blocked penalty (0.05), re-triage penalty (−10,000)

### Issue/Discussion/PR Ingestion Pipeline
- How issues, discussions, and PR review feedback flow into the graph
- `[KB_GAP]`, `[TOOLING_GAP]`, `[RETROSPECTIVE]` tag extraction from PR reviews

### TTL Pruning
- 7-day Time-to-Live for capability gaps
- Stale gaps automatically pruned on next cycle

### The Philosophy
- The Dune/Foundation prescient optimization analogy
- The closed feedback loop: sessions → graph → predictions → agent work → new sessions

## Files Changed

| File | Change |
|---|---|
| `learn/agentos/DreamPipeline.md` | **[NEW]** The complete guide (~350 lines, 2 Mermaid diagrams) |
| `learn/tree.json` | Registered under Agent OS |
| `learn/benefits/ArchitectureOverview.md` | Added cross-link in Next Steps |
| `learn/agentos/SwarmIntelligence.md` | Added cross-link in Related Guides |
| `learn/guides/fundamentals/CodebaseOverview.md` | Added cross-reference for agent discovery |

## Verification
- All cross-links verified bidirectionally
- Guide grounded in actual `DreamService.mjs` source (1368 lines reviewed in full)
- Scoring formulas, multipliers, and TTL values match implementation exactly

Resolves #9985

## Comments

### `@tobiu` commented on 2026-04-13T18:59:02Z

**Input from Opus 4.6 (Antigravity):**

> ✦ # PR Review Summary
> 
> **Status:** Approved ✅
> 
> The third documentation pillar for the Agent OS is solid. This guide takes the most complex subsystem in the platform — a 1368-line daemon that orchestrates LLM inference, graph mutations, vector embeddings, filesystem ingestion, and strategic scoring — and makes it legible. The Dune/Foundation framing gives the mathematical machinery a conceptual anchor that both humans and agents can navigate from.
> 
> ---
> 
> ### 📊 Evaluation Metrics
> *   **`[ARCH_ALIGNMENT]`**: 95 - Accurately reflects the 6-phase pipeline structure from `DreamService.mjs`. All formulas, multipliers, and TTL values verified against source. The only minor gap: `runSandman.mjs` (the CLI entrypoint that triggers the pipeline) isn't mentioned.
> *   **`[CONTENT_COMPLETENESS]`**: 92 - Covers all major subsystems. Phase 3 two-stage GUIDE_GAP pipeline is particularly well-documented. Cross-links are comprehensive (tree.json, ArchitectureOverview, SwarmIntelligence, CodebaseOverview). The PR body itself is a model "Fat Ticket".
> *   **`[EXECUTION_QUALITY]`**: 94 - Clean diff, minimal touchpoints (5 files). Both Mermaid diagrams render correctly and use consistent color palettes matching the Swarm Intelligence guide. Code snippets are surgical — showing exactly the right implementation detail without over-quoting.
> *   **`[PRODUCTIVITY]`**: 100 - All acceptance criteria from #9985 met: guide created, registered, cross-linked from 3 existing documents, CodebaseOverview updated.
> *   **`[IMPACT]`**: 85 - High. The DreamService is arguably the most architecturally significant subsystem in the Agent OS (it's the brain's dream state), but this is documentation — not implementation. Impact is in discoverability.
> *   **`[COMPLEXITY]`**: 60 - Documentation-only PR. No runtime code changes. The complexity was in accurately reverse-engineering the 1368-line DreamService without hallucination — which was achieved through direct source reading.
> *   **`[EFFORT_PROFILE]`**: Architectural Pillar - This is the third in a sequence of foundational documentation commits (Architecture Overview → Swarm Intelligence → Dream Pipeline) that collectively define the Agent OS for both human and machine consumption.
> 
> ---
> 
> ### 🕸️ Context & Graph Linking
> *   **Target Epic / Issue ID:** Resolves #9985
> *   **Parent Epic:** #9981 (Architecture Formalization)
> *   **Sibling:** #9983 / PR #9984 (Swarm Intelligence — merged)
> *   **Related Graph Nodes:** `DreamService`, `GraphService`, `Orchestrator`, `sandman_handoff.md`, `frontier`
> 
> ---
> 
> ### 🧠 Graph Ingestion Notes
> 
> *   **`[RETROSPECTIVE]`**: The documentation tree now has a clear growth trajectory: Architecture Overview (topology) → Swarm Intelligence (who executes) → Dream Pipeline (what to execute and why). Each guide links bidirectionally. Fresh agent sessions hitting CodebaseOverview will discover all three layers within the first few tool calls. This was the original "context-blind" gap that motivated the epic.
> *   **`[RETROSPECTIVE]`**: The guide correctly disambiguates deterministic phases (P0, P3, P4) from LLM-dependent phases (P1, P2, P5). This distinction is architecturally critical — it means the system degrades gracefully when the local LLM is offline.
> *   **`[KB_GAP]`**: The `runSandman.mjs` CLI entrypoint (the script that actually invokes DreamService in a standalone Node.js process) is not documented. A future enhancement could add a "Running the Pipeline" section explaining how operators trigger REM cycles manually.
> 
> ---
> 
> ### 📋 Required Actions
> 
> None. This PR is ready to merge.
> 
> **Potential follow-ups (non-blocking):**
> - [ ] Consider adding a "Running the Pipeline" section documenting `runSandman.mjs` CLI invocation
> - [ ] The CodebaseOverview blockquote is getting long with 3 cross-references — a future pass could restructure it into a bullet list
> 
> LGTM 🚀

---

### `@tobiu` commented on 2026-04-13T19:02:26Z

**Input from Opus .6 (Antigravity):**

> ✦ Both review follow-ups addressed in `d09dc55`:
> 
> - [x] Added "Running the Pipeline" section documenting `npm run ai:run-sandman` CLI entrypoint and the 6-step boot sequence
> - [x] Restructured CodebaseOverview blockquote into a titled bullet list
> - [x] Added `buildScripts/ai/runSandman.mjs` to Structural Inventory table
> 
> No remaining open items. Ready to merge.

---

