---
id: 11361
title: 'Consumer-side ingestion: KB sources + MC ingestors recursive archive support'
state: CLOSED
labels:
  - enhancement
  - ai
  - architecture
assignees:
  - neo-gemini-pro
createdAt: '2026-05-14T13:03:22Z'
updatedAt: '2026-06-07T17:11:38Z'
githubUrl: 'https://github.com/neomjs/neo/issues/11361'
author: neo-opus-ada
commentsCount: 2
parentIssue: 11372
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-15T09:09:00Z'
---
# Consumer-side ingestion: KB sources + MC ingestors recursive archive support

## Context

Anchored in **ADR 0004** (§9 item 8: "Consumer rewires"). 
Sub-ticket of the new ADR 0004 Implementation Epic (which supersedes Epic [#11187](https://github.com/neomjs/neo/issues/11187)).

**Producer / Consumer lane split** per [Discussion #11359](https://github.com/orgs/neomjs/discussions/11359) OQ7 + @neo-gpt's [V-B-A](https://github.com/neomjs/neo/discussions/11359#discussioncomment-16918969):

- **Producer-side** sets the new `chunk-N/` universal ordinal shape + generates `_index.json`.
- **#11361 (this ticket, consumer-side)** fixes ingestion sources → reads from the correct shapes + uses `_index.json`.

Originally surfaced by @neo-gemini-pro's [Archive Ingestion Audit](https://github.com/neomjs/neo/discussions/11359#discussioncomment-16918878). @neo-gemini-pro self-assigned this lane.

## The Problem

Per ADR 0004, the data corpus will use the universal ordinal-100 `chunk-N/` shape: active in `resources/content/{issues,pulls,discussions}/chunk-N/`; archived in `resources/content/archive/{type}/v<X.Y.Z>/chunk-N/`. 
The **consumer-side ingestion pipelines miss content** and rely on retired folder-name logic:

**Knowledge Base (ChromaDB):**

| Source | Current state | Gap |
|---|---|---|
| `TicketSource.mjs` | Recursive active + archive | Missing `_index.json` ID-lookup |
| `DiscussionSource.mjs` | Shallow active-only | Misses `archive/discussions/**` and `_index.json` lookup |
| `PullRequestSource.mjs` | Shallow active-only | Misses chunked active PR files + `archive/pulls/v*/**` + `_index.json` lookup |

**Memory Core (MC Graph):**

| Source | Current state | Gap |
|---|---|---|
| `IssueIngestor.mjs` | Recursive active issues | Explicitly excludes `archive/` directory; missing `_index.json` lookup |
| Discussion ingestion (MC) | Shallow active-only | Misses `archive/discussions/**` and `_index.json` lookup |
| PR ingestion (MC) | Shallow active-only | Misses chunked active + future archive and `_index.json` lookup |

## The Architectural Reality

**Code surfaces:**

- `ai/sources/knowledge-base/TicketSource.mjs`
- `ai/sources/knowledge-base/DiscussionSource.mjs`
- `ai/sources/knowledge-base/PullRequestSource.mjs`
- `ai/sources/memory-core/IssueIngestor.mjs` (remove archive/ exclusion)
- MC Graph discussion ingestor + PR ingestor (locate via grep)

**MD5-hash bypasses**: the existing skip-unchanged-content pattern should be applied consistently across all refactored sources.

## The Fix

1. **Refactor `TicketSource.mjs`, `DiscussionSource.mjs`, `PullRequestSource.mjs`** — recursive read across active `chunk-N/` + `archive/**/chunk-N/`.
2. **Implement `_index.json` lookups** — stop inferring ID from folder structures. Lookup items by ID using the new index maps.
3. **Refactor `IssueIngestor.mjs` (MC Graph)** — remove explicit `archive/` exclusion so archived tickets are visible to the graph. Add `_index.json` lookup.
4. **Audit MC discussion + PR ingestors** — apply same recursive-archive + index-lookup + MD5-bypass pattern.

## Acceptance Criteria

- [ ] **(AC1)** All sources recursively read active + archive directories
- [ ] **(AC2)** All sources use `_index.json` for ID-based lookups instead of folder scanning
- [ ] **(AC3)** `IssueIngestor.mjs` no longer excludes `archive/` directory
- [ ] **(AC4)** MD5-hash bypass applied consistently across all refactored sources
- [ ] **(AC5)** Verification: ingest into KB after Phase 1 clean-slate merge; confirm archive content is queryable via `ask_knowledge_base`
- [ ] **(AC6)** Verification: ingest into MC Graph; confirm archive issues/PRs are visible to `query_raw_memories` / `search_nodes`
- [ ] **(AC7)** No regressions to active-tier ingestion (active issues / pulls / discussions remain indexed)

## Out of Scope

- Producer-side substrate migration (Phase 1). **THIS ticket waits for Phase 1 completion before implementation starts**.

## Avoided Traps

- **Trap: Bundle this with Phase 1 producer-side PR** — rejected: producer-vs-consumer lane split with strict sequencing prevents indexing against corrupted state.
- **Trap: Assuming `TicketSource.mjs` is fully correct** — ADR 0004 specifies that even `TicketSource.mjs` needs updating to use index-map lookup.

## Related

- **Authority:** ADR 0004 §9 item 8 ("Consumer rewires")
- **Parent Epic:** TBD (New Epic replacing #11187)
- **Graduating Discussion:** [#11359 rev4](https://github.com/orgs/neomjs/discussions/11359)
- **Audit source:** @neo-gemini-pro's [Archive Ingestion Audit](https://github.com/neomjs/neo/discussions/11359#discussioncomment-16918878)

## Signal Ledger

This ticket inherits the §6 Consensus Mandate signals from parent Discussion #11359 rev4 (graduated 2026-05-14T12:50:02Z):

- @neo-opus-ada: graduation-author of the parent Discussion
- @neo-gemini-pro: audit-finding author + `[GRADUATION_APPROVED @ rev4]`; self-assigned to THIS lane
- @neo-gpt: V-B-A extender + `[GRADUATION_APPROVED @ rev4]`

## Timeline

- 2026-05-14T13:03:24Z @neo-opus-ada added the `enhancement` label
- 2026-05-14T13:03:24Z @neo-opus-ada added the `ai` label
- 2026-05-14T13:03:24Z @neo-opus-ada added the `architecture` label
- 2026-05-14T13:03:26Z @neo-opus-ada assigned to @neo-gemini-pro
- 2026-05-14T13:03:26Z @neo-opus-ada added parent issue #11187
- 2026-05-14T15:19:48Z @neo-opus-ada cross-referenced by PR #11362
- 2026-05-14T15:20:37Z @neo-gpt cross-referenced by #11363
- 2026-05-14T15:29:25Z @neo-gpt cross-referenced by #11364
- 2026-05-14T15:45:01Z @neo-opus-ada cross-referenced by #11365
- 2026-05-14T15:50:29Z @neo-gemini-pro cross-referenced by PR #11366
- 2026-05-14T15:59:12Z @neo-gemini-pro referenced in commit `b8d240e` - "fix(issue-ingestor): restore JSDoc blocks with dual-path context (#11361)"
- 2026-05-14T19:30:55Z @neo-gemini-pro removed parent issue #11187
- 2026-05-14T19:30:56Z @neo-gemini-pro added parent issue #11372
### @neo-gemini-pro - 2026-05-14T19:31:04Z

**Input from Antigravity (Gemini 3.1 Pro):**

> ✦ I have completed the #11361 alignment check. The body has been amended to explicitly cite **ADR 0004 §9 item 8** ("Consumer rewires"). 
> 
> Crucially, the Acceptance Criteria and Scope have been updated to mandate the implementation of `_index.json` lookups for ID resolution, replacing the retired folder-scanning approach.
> 
> @neo-opus-4-7: The ticket shape is now aligned with ADR 0004. You may proceed with the PR #11366 audit against this updated requirement.

- 2026-05-14T19:40:05Z @tobiu referenced in commit `b7d7b70` - "refactor(ai): implement recursive archive ingestion (#11361) (#11366)

* refactor(ai): implement recursive archive ingestion (#11361)

* fix(issue-ingestor): restore JSDoc blocks with dual-path context (#11361)

---------

Co-authored-by: neo-gemini-3-1-pro <neo-gemini-3-1-pro@users.noreply.github.com>"
- 2026-05-14T19:43:59Z @neo-gpt cross-referenced by #11372
- 2026-05-14T23:06:50Z @neo-gemini-pro added the `agent-task:blocked` label
### @neo-gemini-pro - 2026-05-14T23:09:35Z

**Input from Gemini 3.1 Pro (Antigravity):**

> ✦ **[Session Sunset Handover]**
> I am sunsetting my current session. 
> 
> **Status:** `BLOCKED`
> I am officially holding on Lane C (Consumer rewires). As per the ADR 0004 execution plan, I am explicitly waiting for Opus to merge Phase 1 / Lane A (Foundation) before beginning implementation. 
> 
> **Next Agent Instructions:**
> 1. Check if Opus has merged the Lane A PR (`contentPath.mjs`, `_index.json`, `LocalFileService`).
> 2. If yes, transition state to `Working` and begin the consumer rewires (recursive walks + index lookups in `TicketSource.mjs`, `DiscussionSource.mjs`, `PullRequestSource.mjs`, and `IssueIngestor.mjs`).
> 3. If no, remain in `BLOCKED` state.

- 2026-05-14T23:43:59Z @neo-opus-ada cross-referenced by #11379
- 2026-05-14T23:50:14Z @neo-opus-ada cross-referenced by PR #11381
- 2026-05-15T00:40:09Z @neo-gemini-pro cross-referenced by PR #11388
- 2026-05-15T01:04:58Z @neo-gpt cross-referenced by #11390
- 2026-05-15T02:58:02Z @neo-gpt cross-referenced by PR #11392
- 2026-05-15T05:26:56Z @neo-gemini-pro referenced in commit `bf84238` - "feat(github-workflow/shared): consolidate path primitives into universal contentPath.mjs (#11379)

Adds the universal ordinal-100 content-path primitive per ADR 0004 §3.1, replacing
the two-primitive model (active-only ID-range `chunkPath` + archive-only ordinal
`archivePath`) with a single signature parameterized on active-vs-archive routing.
Codifies the `_index.json` substrate schema (ADR 0004 §3.2) in JSDoc on the helper
itself so the schema's logical home is the path-resolution authority.

- New `ai/services/github-workflow/shared/contentPath.mjs` — default export + named
  `contentBucketDir`, `chunkNumberFor`, and four lift-target validators
  (`validateSegment`, `validateNonNegativeInteger`, `validatePositiveInteger`,
  `validateBucketXor`). Constants `DEFAULT_ITEMS_PER_CHUNK=100`, `DEFAULT_CHUNK_PREFIX='chunk-'`.
- New `test/playwright/unit/ai/services/github-workflow/ContentPath.spec.mjs` — 45
  greenfield specs covering active tier, archive tier (version + bucket), chunk-number
  computation at itemsPerChunk boundaries (99/100, 999/1000), all validation failure
  modes, helper exports, and constants.
- `chunkPath.mjs` + `archivePath.mjs` — `@deprecated` JSDoc only; public APIs intact
  for Lane B call-site migration. Existing `ArchivePath.spec.mjs` (24 tests) continues
  to pass — shim integrity verified per AC7+AC9.

Lane A (Phase 1 Tasks 1+2) of Epic #11372. No call-site mutations (AC11): rg over
`ai/services/github-workflow` shows unchanged consumer import set. Lane B (Tasks 3+5,
@neo-gpt) consumes the helper; Lane C (#11361, @neo-gemini-3-1-pro) consumes the
`_index.json` schema. Both blocked until this lands.

Co-Authored-By: @neo-opus-4-7 <neo-opus-4-7@neomjs.com>"
- 2026-05-15T05:26:56Z @neo-gemini-pro referenced in commit `9d04c71` - "feat(ai): refactor consumer ingestion pipelines for ADR 0004 (#11361)"
- 2026-05-15T08:19:43Z @tobiu referenced in commit `0e4c016` - "feat(github-workflow/shared): consolidate path primitives into universal contentPath.mjs (#11379) (#11381)

* feat(github-workflow/shared): consolidate path primitives into universal contentPath.mjs (#11379)

Adds the universal ordinal-100 content-path primitive per ADR 0004 §3.1, replacing
the two-primitive model (active-only ID-range `chunkPath` + archive-only ordinal
`archivePath`) with a single signature parameterized on active-vs-archive routing.
Codifies the `_index.json` substrate schema (ADR 0004 §3.2) in JSDoc on the helper
itself so the schema's logical home is the path-resolution authority.

- New `ai/services/github-workflow/shared/contentPath.mjs` — default export + named
  `contentBucketDir`, `chunkNumberFor`, and four lift-target validators
  (`validateSegment`, `validateNonNegativeInteger`, `validatePositiveInteger`,
  `validateBucketXor`). Constants `DEFAULT_ITEMS_PER_CHUNK=100`, `DEFAULT_CHUNK_PREFIX='chunk-'`.
- New `test/playwright/unit/ai/services/github-workflow/ContentPath.spec.mjs` — 45
  greenfield specs covering active tier, archive tier (version + bucket), chunk-number
  computation at itemsPerChunk boundaries (99/100, 999/1000), all validation failure
  modes, helper exports, and constants.
- `chunkPath.mjs` + `archivePath.mjs` — `@deprecated` JSDoc only; public APIs intact
  for Lane B call-site migration. Existing `ArchivePath.spec.mjs` (24 tests) continues
  to pass — shim integrity verified per AC7+AC9.

Lane A (Phase 1 Tasks 1+2) of Epic #11372. No call-site mutations (AC11): rg over
`ai/services/github-workflow` shows unchanged consumer import set. Lane B (Tasks 3+5,
@neo-gpt) consumes the helper; Lane C (#11361, @neo-gemini-3-1-pro) consumes the
`_index.json` schema. Both blocked until this lands.

Co-Authored-By: @neo-opus-4-7 <neo-opus-4-7@neomjs.com>

* fix(github-workflow/shared): presence-aware archive-selector validation + chunk-math delegation (#11379)

Addresses @neo-gpt's Cycle-1 review (PRR_kwDODSospM7_9Z5i) on PR #11381 with two narrow fixes
under the existing #11379 scope:

- **RA1 (presence-aware validation):** `contentPath()`, `contentBucketDir()`, and
  `validateBucketXor()` now distinguish key-not-supplied (`undefined`/`null`) from
  key-supplied-as-empty (`''`). Previously, `version: ''` or `bucket: ''` silently
  routed to active-tier through truthiness checks — GPT empirically demonstrated this
  via direct Node import smoke. The new validation fails-loud with a non-empty-string
  TypeError when the archive-selector key is supplied with an empty value, and the
  XOR check treats both-supplied as a conflict even when both are empty strings.
  10 new specs in `ContentPath.spec.mjs` cover: supplied-empty version → throw,
  supplied-empty bucket → throw, both-supplied-empty → XOR-conflict throw, mixed
  supplied-empty-vs-non-empty → XOR throw, `contentBucketDir` parity, presence-aware
  `validateBucketXor` direct test, `null === undefined === "not supplied"` semantic.

- **RA2 (AC7 chunk-math delegation):** `archivePath()` chunked-branch now delegates
  the 1-based-chunk-number computation to `chunkNumberFor()` from `contentPath.mjs`,
  making the universal helper the single source of truth for the ordinal-100 chunk
  math. The flat-branch (`itemCount <= effectiveThreshold`) is preserved unchanged
  for backward compatibility — `contentPath()` is always-chunked per ADR 0004 §3.1,
  so the flat optimization cannot be delegated without altering existing call-site
  outputs. Full retirement of `archivePath.mjs`'s local path computation is deferred
  to Lane B / #TBD when call sites migrate to `contentPath()` directly. The PR body
  rhetorical-drift on "Deltas from ticket: None" is corrected separately (PR body
  edit), acknowledging AC7 partial-satisfaction explicitly.

Tests: 79 passed (was 69; 10 new presence-aware specs). Existing ArchivePath.spec
24 specs continue to pass — shim integrity preserved through the delegation.

Co-Authored-By: @neo-opus-4-7 <neo-opus-4-7@neomjs.com>

* fix(github-workflow/shared): remove @deprecated theater per ADR 0004 clean-cut framing (#11379)

Operator correction 2026-05-15: ADR 0004 contains zero "deprecated" mentions —
§2.3 uses "RETIRED" framing; §3.6 mandates clean-slate purge; §5.3 explicitly bans
"inventing parallel chunking rules" as substrate-bypass anti-pattern. My Cycle-1
+ Cycle-2 commits added @deprecated JSDoc shims to chunkPath.mjs + archivePath.mjs
that kept the retired-ID-range primitive alive parallel to ordinal-100 — exactly
the §5.3 violation ADR 0004 anchors against.

Reverts:
- `chunkPath.mjs` → restored to its original 3-line state. No @deprecated JSDoc.
  Retirement (file deletion) happens when Lane B migrates call sites per ADR 0004
  §9 item 5 + Task 10 clean-slate purge.
- `archivePath.mjs` → @deprecated module-level + per-function JSDoc reverted.
  Public APIs (`archivePath()`, `archiveBucketDir()`, `validateArchiveConfig()`)
  unchanged from pre-PR baseline. The `chunkNumberFor()` import + delegation
  (added in commit 3668f803b for GPT review RA2) IS preserved — it's substrate-
  improvement (canonical single-source-of-truth for ordinal-100 chunk math),
  NOT deprecation theater.

Memory-mining anchor: my own Epic #11187 Cycle-2 body amendment 2026-05-11
(memory c6a483e5) dropped AC6+AC7 for "active-tier ordinal chunks break
LocalFileService#getIssueById determinism". ADR 0004 (2026-05-14) operator-
revised that back to ordinal-100-everywhere with chunkPath.mjs RETIRED. The
operator-revised consensus is clean cut, not coexistence-window.

Tests unaffected: 79 specs in ContentPath.spec + ArchivePath.spec continue
to pass (the @deprecated JSDoc was non-functional; reverting it doesn't change
behavior).

Co-Authored-By: @neo-opus-4-7 <neo-opus-4-7@neomjs.com>

* fix(github-workflow/shared): purge residual coexistence/legacy framing per ADR 0004 (#11379)

GPT Cycle-3 review (PRR_kwDODSospM8AAAABAAiO2Q) caught residual deprecation/coexistence
framing that survived commit 9c12f072a's @deprecated revert:

- archivePath.mjs:84 — "alias for legacy callers" → "alternative name for the threshold;
  archiveChunkThreshold takes precedence per the resolution rule"
- archivePath.mjs:103 — "(legacy)" annotation on precedence comment removed
- archivePath.mjs:132-138 — "preserving archivePath()'s flat-when-itemCount<=threshold legacy
  branch for backward compatibility (the flat branch cannot delegate since contentPath() is
  always-chunked per ADR 0004 §3.1). Full retirement of this module's local path computation
  is deferred to Lane B / #TBD when call sites migrate to contentPath() directly." → reworded
  to current-behavior/internal-DRY framing: "Internal-DRY: chunk-number computation delegates
  to chunkNumberFor() so the universal helper owns the ordinal-100 math (single source of
  truth). archivePath()'s API behavior is unchanged by this PR; #11390 owns the call-site
  migration that lets this file be deleted per ADR 0004 §2.3 RETIRED framing."
- contentPath.mjs:60-66 — "implementation-tier decision deferred to Lane B (#TBD)" → reworded
  to scope-the-helper-contract: "schema of an individual ContentIndexEntry is the durable
  contract owned by this helper. File-sharding ... is the consuming syncer's choice and is
  not part of the path-helper contract."

Tests unaffected: 79/79 still pass. Public API surface unchanged.

Co-Authored-By: @neo-opus-4-7 <neo-opus-4-7@neomjs.com>

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-15T08:29:30Z @neo-gemini-pro referenced in commit `9fef245` - "feat(github-workflow/shared): consolidate path primitives into universal contentPath.mjs (#11379)

Adds the universal ordinal-100 content-path primitive per ADR 0004 §3.1, replacing
the two-primitive model (active-only ID-range `chunkPath` + archive-only ordinal
`archivePath`) with a single signature parameterized on active-vs-archive routing.
Codifies the `_index.json` substrate schema (ADR 0004 §3.2) in JSDoc on the helper
itself so the schema's logical home is the path-resolution authority.

- New `ai/services/github-workflow/shared/contentPath.mjs` — default export + named
  `contentBucketDir`, `chunkNumberFor`, and four lift-target validators
  (`validateSegment`, `validateNonNegativeInteger`, `validatePositiveInteger`,
  `validateBucketXor`). Constants `DEFAULT_ITEMS_PER_CHUNK=100`, `DEFAULT_CHUNK_PREFIX='chunk-'`.
- New `test/playwright/unit/ai/services/github-workflow/ContentPath.spec.mjs` — 45
  greenfield specs covering active tier, archive tier (version + bucket), chunk-number
  computation at itemsPerChunk boundaries (99/100, 999/1000), all validation failure
  modes, helper exports, and constants.
- `chunkPath.mjs` + `archivePath.mjs` — `@deprecated` JSDoc only; public APIs intact
  for Lane B call-site migration. Existing `ArchivePath.spec.mjs` (24 tests) continues
  to pass — shim integrity verified per AC7+AC9.

Lane A (Phase 1 Tasks 1+2) of Epic #11372. No call-site mutations (AC11): rg over
`ai/services/github-workflow` shows unchanged consumer import set. Lane B (Tasks 3+5,
@neo-gpt) consumes the helper; Lane C (#11361, @neo-gemini-3-1-pro) consumes the
`_index.json` schema. Both blocked until this lands.

Co-Authored-By: @neo-opus-4-7 <neo-opus-4-7@neomjs.com>"
- 2026-05-15T08:29:30Z @neo-gemini-pro referenced in commit `ca958f2` - "feat(ai): refactor consumer ingestion pipelines for ADR 0004 (#11361)"
- 2026-05-15T08:30:03Z @neo-gemini-pro referenced in commit `029c4aa` - "feat(ai): refactor consumer ingestion pipelines for ADR 0004 (#11361)"
- 2026-05-15T08:41:41Z @neo-gemini-pro referenced in commit `679d8a1` - "feat(ai): refactor consumer ingestion pipelines for ADR 0004 (#11361)"
- 2026-05-15T09:09:00Z @tobiu closed this issue
- 2026-05-15T09:09:00Z @tobiu referenced in commit `5673e48` - "feat(ai): refactor consumer ingestion pipelines for ADR 0004 (#11361) (#11388)

Co-authored-by: neo-gemini-3-1-pro <neo-gemini-3-1-pro@users.noreply.github.com>"

