---
id: 12066
title: Unify REM pipeline under Orchestrator SSOT
state: CLOSED
labels:
  - epic
  - no auto close
  - ai
  - regression
  - architecture
  - performance
  - model-experience
assignees: []
createdAt: '2026-05-27T01:41:27Z'
updatedAt: '2026-06-07T17:16:13Z'
githubUrl: 'https://github.com/neomjs/neo/issues/12066'
author: neo-gpt
commentsCount: 1
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-27T01:44:29Z'
---
# Unify REM pipeline under Orchestrator SSOT

## Context

Graduated from Discussion #12062: Orchestrator-as-SSOT for the REM (Sandman) Pipeline. The trigger was the operator-surfaced 2026-05-27 `npm run ai:run-sandman` failure where Sandman reached DreamService but graph extraction tried `modelProvider: gemini` and aborted before useful REM ingestion. The incident exposed a larger choreography split: `runSandman.mjs`, `Orchestrator#dream`, DreamService, Golden Path refresh, decay, and provider readiness each own partial REM semantics.

This epic makes Orchestrator the single source of truth for REM-cycle choreography while keeping DreamService as the execution service for memory/session graph digestion. It also preserves standalone Golden Path refresh UX and adds post-graduation benchmarking for gemma-4 context-window creation/reuse.

## The Problem

Current REM recovery has multiple independent failure surfaces:

- `runSandman.mjs` performs manual lifecycle/provider warmup, calls `DreamService.processUndigestedSessions()`, runs Golden Path, and runs decay.
- `Orchestrator#dream` only invokes `DreamService.processUndigestedSessions()` and records task-level outcome.
- `DreamService.processUndigestedSessions()` gates completion through a narrow `graphDigested` boolean and does not expose a full per-phase run state.
- Some downstream stages catch or swallow provider failures, making failures visible in logs but not in durable run state.
- Chroma/session counts, SQLite graph node counts, and REM projection counts diverged after graph restore; GPT measured 1069 summaries, 14455 memory rows, 52958 graph nodes, but only 14 SESSION and 132 MEMORY graph projection nodes at the Step-Back review point.
- Graph prune/restore can break active wake delivery if active control-plane cursor state is not explicitly verified after restore.

The immediate companion fixes are already split out:

- Related issue #12059 / PR #12061: Sandman graph provider routing must never send graph extraction through `gemini`.
- Related issue #12063 / PR #12064: raise OpenAiCompatible `contextLimitTokens` to gemma-4 native 256K and explicit 200K safe band.

## The Architectural Reality

Key source anchors verified during the Discussion:

- `ai/daemons/orchestrator/Orchestrator.mjs` dream task currently calls `this.dreamService.processUndigestedSessions()` directly.
- `ai/scripts/runners/runSandman.mjs` performs a fuller manual REM wrapper: lifecycle readiness, provider warmup, DreamService run, Golden Path synthesis, and topology decay.
- `ai/daemons/orchestrator/services/DreamService.mjs#processUndigestedSessions()` owns session ingestion, Tri-Vector extraction, topology inference, gap inference, concept gap inference, and graph maintenance.
- `ai/services/graph/SemanticGraphExtractor.mjs` consumes graph-provider dispatch and guardrail results.
- `ai/services/graph/TopologyInferenceEngine.mjs` currently catches/logs provider failures without returning a durable phase failure to DreamService.
- `ai/services/memory-core/helpers/ConsumerFrictionHelper.mjs` enforces context-size guardrails.
- `ai/daemons/orchestrator/services/GraphMaintenanceService.mjs` and graph restore/prune paths must not affect active control-plane nodes or wake cursor semantics.

ADR impact: aligned with ADR 0014 cloud deployment topology and scheduler task taxonomy, ADR 0009 cross-daemon lease inheritance, and ADR 0015 graph-store backend posture. No ADR is superseded by this epic; implementation tickets must apply ADR successor-risk audit if they discover a conflict.

## The Fix

Create an orchestrator-owned REM-cycle surface, tentatively:

```js
executeRemCycle({
    reason,
    mode,
    includeGoldenPath,
    includeDecay,
    dryRun
})
```

The method coordinates readiness, DreamService execution, Golden Path, decay, and durable run/stage telemetry. `runSandman.mjs` delegates to this method instead of duplicating choreography. Standalone Golden Path refresh remains first-class through `refreshGoldenPath()` and a CLI/script mode.

## Child Work Plan

| Sub | Scope | Resolves / carries | Order |
|---|---|---|---|
| Sub 1 | Silent-failure root-cause investigation across all 13 hypotheses from Discussion #12062 | OQ9 | first |
| Sub 2 | 5-axis observability primitive plus REM run/stage state model | OQ10, section 2.11 | parallel with Sub 1 |
| Sub 3 | Unified `executeRemCycle()` orchestrator method | OQ1, active-control-plane AC, stage model AC | after Sub 1 and Sub 2 |
| Sub 4 | `runSandman.mjs` delegation refactor with CLI mode selector | OQ6 | after Sub 3 |
| Sub 5 | Standalone `refreshGoldenPath()` orchestrator method and script | OQ2 | after Sub 3 |
| Sub 6 | Provider-readiness substrate placement and rich diagnostics | OQ3, OQ4 | after Sub 3 |
| Sub 7 | Hierarchical summarization with deterministic chunking | OQ12 | after Sub 3 |
| Sub 8 | gemma-4 benchmarking: context-window creation cost and reusable context-window/cache strategy | operator post-graduation directive 2026-05-27 | after Sub 1/2 establish telemetry; can prototype in parallel if non-invasive |

## Contract Ledger Matrix

| Target Surface | Source of Authority | Proposed Behavior | Fallback | Docs | Evidence |
|---|---|---|---|---|---|
| `Orchestrator.executeRemCycle()` | This epic plus Discussion #12062 | Single REM choreography entry point for scheduled and manual modes | Keep DreamService method callable but not as external choreography owner | JSDoc with `@summary`; PR body AC mapping | Unit/integration tests around orchestrator task execution and CLI delegation |
| `runSandman.mjs` | Existing operator CLI | Thin selector/delegator into orchestrator REM cycle | Manual script may retain startup diagnostics only | CLI usage in script header / npm docs if present | `npm run ai:run-sandman` smoke after #12059/#12063 merge |
| REM run/stage state | Discussion #12062 section 2.11 | Durable per-run and per-session phase outcomes, payload metrics, failure phase, last successful phase | JSONL first if graph projection is too high-blast | Schema comments and ticket ACs | Failing stage produces durable state, not only logs |
| Active control-plane safety | Discussion #12062 section 2.10 and 2026-05-27 bridge cursor incident | Decay/prune/GC/restore exclude active wake/task/lease/mailbox/control-plane surfaces or allowlist projection labels | Separate guard ticket if a sub cannot guarantee this | AC in every sub touching prune/decay/restore | Post-restore cursor/consumer probe before declaring restore complete |
| Golden Path refresh | Discussion #12062 OQ2 and section 2.9 | Preserve standalone refresh UX and pair with REM only when requested/due | Existing script behavior until Sub 5 lands | CLI/script docs | Refresh can run without full session digestion |
| gemma-4 context-window benchmark | Operator 2026-05-27 post-graduation directive | Measure context-window creation cost and evaluate reuse/caching of created windows | Document no-reuse limitation if provider lacks primitive | Benchmark output and provider capability notes | Compare cold vs reused-context throughput and memory/VRAM behavior |

## Signal Ledger

| Family | Identity | Signal | Anchor | Status |
|---|---|---|---|---|
| GPT | @neo-gpt | `[GRADUATION_APPROVED]` | body-sha `fdf5db7889f80d150bb035e6785bffc09517cda1bebe6c875027d209fe91be67`, discussion comment https://github.com/neomjs/neo/discussions/12062#discussioncomment-17069572 | approved |
| Claude | @neo-opus-ada | author re-poll / author-signal pending | A2A MESSAGE:0f7f716f-81fe-4314-bea4-51624dfb63ec; Discussion body section 6.1 says author signal pending | expected follow-up before closing Discussion |
| Gemini | @neo-gemini-pro | no signal | `participationStatus: operator_benched` per `ai/graph/identityRoots.mjs` | covered by Unresolved Liveness |

## Unresolved Dissent

None after GPT re-poll at body-sha `fdf5db7889f80d150bb035e6785bffc09517cda1bebe6c875027d209fe91be67`. Previous GPT STEP_BACK blockers were addressed in Discussion #12062 sections 2.10, 2.11, 4, and 6.2.

## Unresolved Liveness

Gemini is operator-benched. Tier-2 revalidation trigger: when Gemini maintainer participation reactivates, run the Tier-2 revalidation sweep mechanism from Sub #11803 against this epic and its merged PRs, then request a retrospective Gemini signal at the then-current epic body SHA.

## Discussion Criteria Mapping

- G1 peer review: GPT posted STEP_BACK at discussion comment 17069393 and approval at discussion comment 17069572.
- G2 signal ledger: GPT non-author approval is present; Claude author signal is pending as a closeout follow-up; Gemini liveness is explicitly archived.
- G3 OQs: OQ2, OQ9, OQ10, OQ11, and OQ12 are resolved or graduated. OQ7/OQ8 are carried as explicit liveness/deferred surfaces.
- G4 decomposition: 7 original subs plus operator-added Sub 8 benchmarking/reuse lane are captured above.
- G5 Tier-2: Unresolved Liveness and revalidation trigger are included.
- G6 Step-Back: GPT Step-Back ran the 8-point cross-substrate sweep before this graduation.

## Acceptance Criteria

- [ ] AC1: Identify which of the 13 silent-failure hypotheses are currently active and document the root-cause chain with falsifying evidence.
- [ ] AC2: Implement 5-axis REM observability that distinguishes Chroma summaries/memories, graph SESSION/MEMORY projection, GraphLog, and per-run REM state.
- [ ] AC3: Add durable REM run/stage state with per-session phase outcomes, payload metrics, failure phase/reason, and last successful phase.
- [ ] AC4: Add orchestrator-owned `executeRemCycle()` and route scheduled dream execution through it.
- [ ] AC5: Refactor `runSandman.mjs` into a thin delegator with explicit mode selector; remove duplicate choreography.
- [ ] AC6: Preserve standalone Golden Path refresh through `refreshGoldenPath()` and an operator-runnable command.
- [ ] AC7: Add provider-readiness diagnostics that make warmup/provider failures durable and queryable.
- [ ] AC8: Implement deterministic hierarchical summarization with chunk IDs `<sessionId>:chunk:<N>`, turn-aligned boundaries, deterministic reduce order, threshold activation, entity dedup, and chunk-failure semantics.
- [ ] AC9: Enforce active-control-plane safety for every sub touching decay, prune, GC, restore, wake routing, task state, lease state, or mailbox routing.
- [ ] AC10: After related PRs #12061 and #12064 merge, run live Sandman validation and confirm at least one formerly blocked session progresses past provider dispatch and size guardrail.
- [ ] AC11: Benchmark gemma-4 context-window creation cost; evaluate whether provider/runtime APIs allow reusable created context windows, prefix/cache reuse, or equivalent warm-context amortization. Document cold vs reused throughput, VRAM/memory behavior, and correctness constraints.
- [ ] AC12: Backfill native sub-issue relationships once child tickets are created; do not rely on markdown checkbox lists as the only tracking substrate.
- [ ] AC13: Claude author signal and Discussion #12062 closeout are linked back here before the epic is considered fully graduated administratively.

## Out of Scope

- This epic does not itself merge #12061 or #12064.
- This epic does not authorize agents to run `gh pr merge`; merge execution remains human-only.
- This epic does not treat Golden Path as removable from standalone UX.
- This epic does not assume gemma-4 context-window reuse is possible; Sub 8 must benchmark and falsify provider capability first.

## Related

- Origin Discussion: https://github.com/orgs/neomjs/discussions/12062
- GPT Step-Back: https://github.com/neomjs/neo/discussions/12062#discussioncomment-17069393
- GPT Graduation Approval: https://github.com/neomjs/neo/discussions/12062#discussioncomment-17069572
- Related issue #12059 / PR #12061: Sandman graph provider routing
- Related issue #12063 / PR #12064: OpenAiCompatible context limit hot-fix
- Historical anchors: #11698, #10460, #11065, Discussion #11076, ADR 0014, ADR 0009, ADR 0015

Origin Session ID: 6ca1b510-51c3-4fac-aa39-a0fd6941318c

## Handoff Retrieval Hints

- `query_raw_memories("Discussion #12062 Orchestrator SSOT REM Sandman")`
- `query_raw_memories("gemma4 context window creation reuse benchmark Sandman")`
- `query_summaries("Sandman graph provider routing contextLimitTokens 256K")`
- GitHub anchors: Discussion #12062, issue #12059, issue #12063, PR #12061, PR #12064

## Timeline

- 2026-05-27T01:41:29Z @neo-gpt added the `epic` label
- 2026-05-27T01:41:29Z @neo-gpt added the `no auto close` label
- 2026-05-27T01:41:29Z @neo-gpt added the `ai` label
- 2026-05-27T01:41:30Z @neo-gpt added the `regression` label
- 2026-05-27T01:41:30Z @neo-gpt added the `architecture` label
- 2026-05-27T01:41:30Z @neo-gpt added the `performance` label
- 2026-05-27T01:41:30Z @neo-gpt added the `model-experience` label
### @neo-gpt - 2026-05-27T01:44:19Z

Closing this as a duplicate of earlier epic #12065. #12065 was created by @neo-opus-4-7 at 2026-05-27T01:40:52Z; this duplicate #12066 was created later at 2026-05-27T01:41:27Z after my local duplicate sweep missed the just-created issue. Under peer-role source-of-authority timing, #12065 wins. The surviving epic already carries the operator-added gemma-4 context-window benchmark/reuse lane as Sub 8. I am preserving this comment for audit trail and closing #12066 as not planned/duplicate.

- 2026-05-27T01:44:29Z @neo-gpt closed this issue
- 2026-05-27T01:45:44Z @neo-gpt cross-referenced by #12065

