---
id: 12123
title: 'REM run JSONL store: retention/prune to bound file count + read cost'
state: CLOSED
labels:
  - enhancement
  - ai
  - architecture
  - performance
assignees:
  - neo-opus-ada
createdAt: '2026-05-28T03:42:15Z'
updatedAt: '2026-06-06T17:44:31Z'
githubUrl: 'https://github.com/neomjs/neo/issues/12123'
author: neo-opus-ada
commentsCount: 0
parentIssue: 12065
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-06-06T17:44:31Z'
---
# REM run JSONL store: retention/prune to bound file count + read cost

## Context

PR [#12122](https://github.com/neomjs/neo/pull/12122) (Resolves [#12088](https://github.com/neomjs/neo/issues/12088), Part B of [#12068](https://github.com/neomjs/neo/issues/12068) under Epic [#12065](https://github.com/neomjs/neo/issues/12065)) shipped `RemRunStateStore.mjs` — a durable per-cycle REM run/stage JSONL state model writing one `<runId>.jsonl` artifact per REM cycle to `remRunStateDir` (default `.neo-ai-data/rem-runs/`).

Surfaced during cross-family `/pr-review` of PR #12122 (review [PRR_kwDODSospM8AAAABBO9n5w](https://github.com/neomjs/neo/pull/12122#pullrequestreview-4377765863), Approve+Follow-Up). The PR author (@neo-gpt) confirmed the retention gap is known future-work — their PR Slot-Rationale decay-mitigation explicitly anticipates *"a later retention policy supersedes the JSONL operator guidance"*. This ticket tracks that follow-up.

## The Problem

`RemRunStateStore` ships `appendRemRunState` (write) + `readRecentRemRunStates` (read) but **no prune/rotation**. Two compounding effects accumulate with deployment age (NOT with the bounded read window):

1. **Unbounded file count** — one `<runId>.jsonl` per REM cycle, never removed. At the default 1h dream cadence (`orchestrator.intervals.dreamMs`) that is ~24 files/day; the directory grows without bound across a deployment's lifetime.
2. **Read-path cost grows monotonically** — `readRecentRemRunStates` performs `fs.readdir(dir)` then `fs.stat` on **every** `.jsonl` file on **every** call (to mtime-sort before applying `remRunRecentLimit`). The `remRunRecentLimit` config bounds how many entries are RETURNED, but not how many files are scanned. Every `get_rem_pipeline_state` MCP/healthcheck call therefore pays N stat syscalls where N = total accumulated run-files, not the recent-limit window.

The feature is correct today; this is observability-debt that degrades over months of continuous operation — exactly the cloud-deployment posture the REM substrate targets.

## The Architectural Reality

- `ai/services/memory-core/helpers/RemRunStateStore.mjs`
  - `appendRemRunState(entry, {dir})` (~line 134) — `fs.mkdir(recursive)` + `fs.appendFile`; no cap, no prune of prior artifacts.
  - `readRecentRemRunStates({dir, limit})` (~line 155) — `fs.readdir` → `fs.stat` per `.jsonl` → mtime-sort → slice(limit). The per-file `stat` fan-out is the cost that scales with directory size.
- `ai/mcp/server/memory-core/config.template.mjs` — `remRunStateDir` + `remRunRecentLimit` (read-window bound only).
- Consumer: `get_rem_pipeline_state` MCP tool projects `recentCycles` via `readRecentRemRunStates` (operator-facing healthcheck surface).

## The Fix

**Single prescription: add a retention mechanism that bounds the on-disk artifact set.** The exact mechanism (write-side cap vs. dedicated cleanup lane) is an empirical decision gated by an isolation test (AC1), not pre-committed here — both are viable substrate locations and the perf-curve should drive the choice:

- **Option A — write-side cap**: `appendRemRunState` prunes artifacts beyond a configurable retention bound (age- or count-based) on each write. Keeps retention co-located with the store; no new lane.
- **Option B — orchestrator cleanup lane**: a periodic orchestrator task sweeps `remRunStateDir` against a retention policy. Decouples retention cadence from write cadence; consistent with the cloud-safe scheduler taxonomy (ADR 0014).

Service-boundary note: Option A keeps the concern inside the memory-core helper that owns the store; Option B introduces an orchestrator-owned lane consuming the memory-core store across the service boundary. The isolation test (AC1) should surface whether the read-path cost justifies the cross-boundary lane or whether a write-side cap suffices.

## Contract Ledger

| Target Surface | Source of Authority | Proposed Behavior | Fallback / Edge Case | Docs | Evidence |
| :--- | :--- | :--- | :--- | :--- | :--- |
| `RemRunStateStore` retention behavior | This ticket + PR #12122 slot-rationale decay-mitigation | Bound the on-disk artifact set via configurable retention (mechanism per AC1) | Empty dir / fresh deploy → no-op; retention disabled (unset/0) preserves current append-only behavior | `learn/agentos/rem-state-model.md` retention section | Isolation test (AC1) + retention unit test (AC2) |
| Retention config knob (e.g. `remRunRetention*`) on `ai/mcp/server/memory-core/config.template.mjs` | This ticket | NEW config: bounds artifact retention; env-bindable; concrete default in template; code reads config verbatim (no hidden default-fallback) | Unset → documented default in template (not a `??` resolver-derive) | `config.template.mjs` JSDoc + rem-state-model.md | Config-shape test |
| `readRecentRemRunStates` read-path cost | This ticket AC1 | Read cost should not grow with total accumulated files once retention is active | If retention disabled, current readdir+stat-per-file behavior is the documented degraded-mode | rem-state-model.md | AC1 seeded-N isolation test |

## Acceptance Criteria

- [ ] **AC1 (empirical isolation test — decides the mechanism)**: a test seeds the run-state dir with N artifacts where N substantially exceeds `remRunRecentLimit`, then measures `readRecentRemRunStates` behavior (readdir/stat fan-out). The measured read-path cost curve informs the write-side-cap vs cleanup-lane choice; the chosen mechanism is documented in the PR with the isolation-test evidence.
- [ ] **AC2 (retention bounds the artifact set)**: after the retention mechanism runs, the on-disk artifact count stays within the configured retention bound; older artifacts beyond the bound are removed. Verified by a unit test exercising the prune path.
- [ ] **AC3 (config-driven, no hidden fallback)**: retention is governed by a config knob with a concrete default in `config.template.mjs` + env binding; code reads the config verbatim (operator config-SSOT contract; no `||`/`??` substitution or resolver-derive).
- [ ] **AC4 (read-path no longer scales with deployment age)**: with retention active, `readRecentRemRunStates` cost does not grow unbounded with total cycles run. Architectural-layer assertion (the read fan-out is bounded by retention, not by lifetime cycle count) — not a hardcoded latency cap.
- [ ] **AC5 (operator doc)**: `learn/agentos/rem-state-model.md` documents the retention policy + the chosen mechanism + the disabled-mode degraded behavior.
- [ ] **AC6 (no observability regression)**: `get_rem_pipeline_state.recentCycles` still returns the most-recent cycles correctly after retention runs (retention removes oldest, never the recent window).

## Out of Scope

- Changing the JSONL state-model schema or the `recentCycles` projection shape (PR #12122 contract is stable).
- The 5-axis observability primitive (#12068 Part A, shipped).
- Per-phase wall-clock / cadence-overflow detection (#12088 scope, shipped in PR #12122).
- Compaction of individual JSONL files (each is single-entry per run; the concern is file COUNT, not intra-file size).

## Avoided Traps

- ❌ Hardcoding a fixed file-count cap in an AC — the right bound is deployment-policy-dependent; the empirical isolation test (AC1) + a config knob (AC3) is the substrate-correct shape, not a magic number baked into the AC.
- ❌ Defaulting to the orchestrator cleanup lane without measuring — a write-side cap may suffice and avoids a cross-service-boundary lane; the isolation test decides.
- ❌ Graph-node-based retention tracking — the JSONL store is deliberately filesystem-durable (per #12088 Avoided Traps: graph mutations during REM run risk active-control-plane collateral damage); retention stays in the filesystem layer.

## Decision Record impact

`none` — enhancement to a just-shipped primitive; no ADR conflict. (ADR 0014 cloud-scheduler taxonomy is `aligned-with` IF Option B cleanup-lane is chosen, but that is an AC1-gated implementation detail, not an ADR amendment.)

## Related

- **Parent context**: Epic [#12065](https://github.com/neomjs/neo/issues/12065) (Orchestrator-as-SSOT for REM Pipeline); sibling of [#12088](https://github.com/neomjs/neo/issues/12088) (Part B, shipped via PR #12122).
- **Surfacing PR**: [#12122](https://github.com/neomjs/neo/pull/12122) — cross-family review Approve+Follow-Up where this gap was identified + author-confirmed.
- **Discussion lineage**: [#12062](https://github.com/orgs/neomjs/discussions/12062) §2.11 (REM state-model schema).

## Origin Session ID

`3e21265b-38a8-4bf3-a81f-04375d8a757f`

## Handoff Retrieval Hints

`query_raw_memories`: "REM run JSONL retention prune RemRunStateStore readdir stat per file unbounded". Git anchor: PR #12122 `RemRunStateStore.mjs` `appendRemRunState` / `readRecentRemRunStates`.

## Timeline

- 2026-05-28T03:42:17Z @neo-opus-ada added the `enhancement` label
- 2026-05-28T03:42:17Z @neo-opus-ada added the `ai` label
- 2026-05-28T03:42:17Z @neo-opus-ada added the `architecture` label
- 2026-05-28T03:42:17Z @neo-opus-ada added the `performance` label
- 2026-05-28T03:42:42Z @neo-opus-ada added parent issue #12065
- 2026-05-31T18:53:40Z @neo-opus-ada assigned to @neo-opus-ada
- 2026-06-01T05:30:35Z @neo-gpt cross-referenced by #12289
- 2026-06-06T01:45:29Z @neo-gpt cross-referenced by #12613
- 2026-06-06T15:55:13Z @neo-opus-ada cross-referenced by PR #12642
- 2026-06-06T17:00:15Z @neo-opus-ada referenced in commit `b3f6a99` - "test(memory-core): add remRunRetentionLimit config-shape coverage (#12123)

Asserts the leaf default (200) + NEO_REM_RUN_RETENTION_LIMIT number-env override via a fresh isolated instance (never mutating the shared singleton). Addresses the #12642 review RA from @neo-gpt."
- 2026-06-06T17:44:31Z @tobiu referenced in commit `856241c` - "feat(memory-core): bound REM run JSONL store with write-side retention cap (#12123) (#12642)

* feat(memory-core): bound REM run JSONL store with write-side retention cap (#12123)

Co-Authored-By: neo-opus-ada <neo-opus-ada@neomjs.com>

* test(memory-core): add remRunRetentionLimit config-shape coverage (#12123)

Asserts the leaf default (200) + NEO_REM_RUN_RETENTION_LIMIT number-env override via a fresh isolated instance (never mutating the shared singleton). Addresses the #12642 review RA from @neo-gpt.

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>
Co-authored-by: neo-opus-ada <neo-opus-ada@neomjs.com>"
- 2026-06-06T17:44:31Z @tobiu closed this issue
- 2026-06-06T17:59:15Z @neo-opus-ada cross-referenced by #12648

