---
number: 10105
title: >-
  feat(ai): ConceptDiscoveryService + validated flag for mined candidates
  (#10036)
author: tobiu
state: MERGED
createdAt: '2026-04-19T16:05:12Z'
updatedAt: '2026-04-19T16:30:21Z'
closedAt: '2026-04-19T16:30:18Z'
mergedAt: '2026-04-19T16:30:18Z'
head: agent/10036-memory-core-concept-discovery
base: dev
url: 'https://github.com/neomjs/neo/pull/10105'
---
## Summary

Lands deliverables **1**, **2**, and **4** of #10036 via **LLM-driven concept extraction** (gemma4 via `OpenAiCompatible.generate` — same substrate used by `SemanticGraphExtractor` and `GoldenPathSynthesizer`). Two high-signal corpora: epic-labeled GitHub issues and recent pull-request markdown (including their `## Comments` sections, where Gold Standards / Traps Avoided / Retrospective prose lives). Deliverables 3 (KB-assisted — timeout-blocked) and 5 (glossary gap report) deferred.

## The Problem

Today's concept ontology (~60 rows) is seeded almost entirely from guide titles. Architectural vocabulary emerging from actual Swarm work — epics, PR reviews — is invisible to the ontology. Without a discovery channel, the only growth path is manual `nodes.jsonl` edits.

#10036 is a semantic-judgment task: *does this term pass the Teaching Test?* That requires reasoning about meaning, not surface patterns. PR review comments in particular (marked with `[ARCH_ALIGNMENT]`, `[RETROSPECTIVE]`, `[KB_GAP]`, or living inside "Gold Standards / Traps Avoided" bullets) are where architectural discourse consolidates — exactly the corpus where new vocabulary emerges.

## Architectural Reality (verified against precedent)

- `ai/daemons/services/SemanticGraphExtractor.mjs` + `GoldenPathSynthesizer.mjs` — established pattern: construct `OpenAiCompatible` via `Neo.create`, call `provider.generate(prompt)`, parse with `Json.extract` (fence-tolerant). Same pipeline.
- `ai/daemons/services/ConceptIngestor.mjs` — additive one-way sync from curated JSONL → graph. `ConceptDiscoveryService` is the reverse-flow proposal channel (JSONL ← discovery); directional invariant preserved.
- `ai/daemons/services/GapInferenceEngine.mjs:inferConceptGraphGaps` — the weight gate already silences low-weight concepts. Adding `if (concept.properties?.validated === false) continue` at the top of the loop is a one-line explicit override that stays orthogonal. Legacy rows without the field (`undefined`) are treated as validated — no data migration.
- `resources/content/pulls/*.md` — 296 PR files today, each with body + `## Comments` section. Sorted descending by number so most-recent PRs (freshest architectural discourse) process first.
- `ConceptService.resolveAlias` + `ConceptService.nodes` — case-insensitive dedupe at the candidate level.

## Implementation Footprint

| File | Change |
|---|---|
| `ConceptDiscoveryService.mjs` (new) | Singleton daemon service. LLM-driven extraction via `OpenAiCompatible.generate`. Teaching-Test system prompt + explicit anti-patterns (class names, file paths, framework names without architectural weight). `mineFromEpics` + `mineFromPullRequests` + `runDiscoveryCycle` + `appendCandidates`. PR cap = 20 most-recent per cycle (`prScanLimit`). Candidates serialize to `nodes.jsonl` with `validated: false`, tier 3, source + reasoning tags. |
| `ConceptIngestor.mjs` | Adds `validated` to payload hash + upsert properties. Curator state-flip triggers re-upsert. |
| `GapInferenceEngine.mjs` | Adds `if (concept.properties?.validated === false) continue` gate. Silences all three signals (GUIDE_GAP / EXAMPLE_GAP / ORPHAN_CONCEPT) for unvalidated candidates regardless of weight. |
| `DreamService.mjs` | Adds `runConceptDiscovery()` facade. Not wired into REM-cycle — deliberate (LLM cost); curator/operator invokes manually. |
| `ConceptDiscoveryService.spec.mjs` (new) | 6 tests: epic-label filtering, PR cap + recency, ConceptService dedupe (id + alias), malformed-LLM tolerance, MIN_SOURCE_LENGTH short-circuit, end-to-end `runDiscoveryCycle` + JSONL append. Mocks `OpenAiCompatible.prototype.generate` with queued responses. Symmetric afterEach per `feedback_symmetric_spec_cleanup.md`. |
| `DreamService.spec.mjs` | New test: `validated: false` silences all three concept-graph signals even with weight 1.3 > 0.8 threshold. |

## Design Decisions

- **Corpora**: epics (curated architecture) + PRs (including comments — where `[ARCH_ALIGNMENT]` / `[RETROSPECTIVE]` / Gold-Standards prose lives). Session-summary mining deferred — epics + PRs are higher signal-to-noise.
- **Single JSONL, no staging file**: per the intake discussion, dual-file would duplicate the curator's review surface. Git diff on `nodes.jsonl` is the review UI. Weight gate + validated flag silence unvalidated rows in the handoff.
- **Manual invocation only**: running discovery every REM cycle would flood `nodes.jsonl` with LLM churn. `DreamService.runConceptDiscovery` is a facade; operator decides cadence.
- **PR cap = 20**: 296 PRs in the repo today. Processing all per cycle would burn budget. Recency-biased (highest PR number first) catches fresh discourse. Configurable via instance `prScanLimit`.
- **MIN_SOURCE_LENGTH = 200**: tiny bodies aren't worth the LLM call. Short-circuit before invoking the provider.
- **Fence-tolerant parsing**: some models emit ```` ```json ```` despite explicit anti-fence instructions. `Json.extract` handles both.

## Gold Standards Leveraged / Traps Avoided

- **Gold Standard** (in-codebase LLM pattern): mirrors `SemanticGraphExtractor` + `GoldenPathSynthesizer` — `Neo.create(OpenAiCompatible, {model, host})` → `provider.generate(prompt)` → `Json.extract(result.content)`.
- **Gold Standard** (additive ingestor pattern): `ConceptDiscoveryService` appends to the same JSONL `ConceptIngestor` reads. Next cycle auto-picks up new candidates; no special routing.
- **Gold Standard** (weight-gate reuse): mined candidates land with tier 3 → default weight 0.3 → below `guideGapWeightThreshold = 0.8` → silenced. `validated: false` is belt-and-suspenders.
- **Trap avoided** (substrate-for-semantic-tasks mismatch): first draft of this PR used PascalCase regex + stop-phrase blocklist. Every individual piece was defensible; the aggregate was still surface-pattern matching for a semantic-judgment task. No stop-phrase list can rescue "Pull Request Review" (noise) vs. "Concept Ontology Layer" (signal) at the regex level. LLM reasoning is the correct substrate. See `feedback_llm_substrate_for_semantic_tasks.md` — surfaced during tobi's self-review challenge.
- **Trap avoided** (dual-file staging): per intake, `candidates.jsonl` split would duplicate curator review surface.
- **Trap avoided** (REM-cycle integration): would flood the JSONL with per-cycle LLM churn. Manual invocation respects cost.

## Testing

```bash
npm run test-unit -- test/playwright/unit/ai/services/ConceptService.spec.mjs test/playwright/unit/ai/daemons/
```

**48 passed** (ConceptService 23, DreamService 12 — includes new validated-gate test, DreamServiceGoldenPath 1, ConceptIngestor 7, ConceptDiscoveryService 6 new). Stable across 5 consecutive runs.

## Out of Scope / Follow-Up

- **Deliverable 3 (KB-assisted classification)** — blocked on `ask_knowledge_base` timeout per ticket body. File follow-up when unblocked.
- **Deliverable 5 (glossary gap report)** — pure reporting tool. File when the concept graph matures.
- **Session-summary mining** — epics + PRs are higher signal; add if coverage proves insufficient.
- **REM-cycle auto-invocation** — manual-only today; add cadence/budget control if curator wants automation.

## Related

- Parent epic: #10030 (Concept Ontology & Semantic Gap Inference)
- Preceded by: #10085 (#10100), #10087 (#10101), #10086 (#10102) — all merged this session
- Depends on: #10035 (concept graph integration — merged as part of #10084)
- Durable memory surfaced: `feedback_llm_substrate_for_semantic_tasks.md`, `feedback_symmetric_spec_cleanup.md`, `feedback_observability_consumer_identity.md`, `feedback_config_service_boundary.md`, `feedback_challenge_prescribed_fixes.md`, `feedback_no_title_prefix_duplicating_labels.md`

Resolves #10036 (deliverables 1+2+4; deliverables 3+5 deferred with explicit rationale)

Origin Session ID: 1db25bbe-f39d-4dcd-bb0e-bc125ce91326

## Comments

### `@tobiu` commented on 2026-04-19T16:21:59Z

**Input from Claude Opus 4.7 (Claude Code):**

> ❋ ## Force-pushed: regex → LLM extraction
> 
> First draft used PascalCase regex + stop-phrase blocklist — architecturally wrong for a semantic-judgment task (catches "Pull Request Review" as readily as "Concept Ontology Layer"; no stop-phrase list is complete enough to rescue it). Tobi's challenge was correct.
> 
> **Replaced with `OpenAiCompatible.generate` (gemma4 via aiConfig)** — same pipeline `SemanticGraphExtractor` and `GoldenPathSynthesizer` already use. System prompt encodes the Teaching Test + explicit anti-patterns.
> 
> **Corpora per tobi's guidance:**
> - **Epics** (`resources/content/issues/*.md`, filtered to `epic` label)
> - **Pull requests** (`resources/content/pulls/*.md`, including `## Comments` sections — where `[ARCH_ALIGNMENT]` / `[RETROSPECTIVE]` / Gold Standards / Traps Avoided prose lives). Recency-biased (sort descending by PR number), capped at 20 most-recent per cycle to bound LLM cost.
> 
> **Scaffolding preserved** (was already architecturally correct): `validated` flag in schema + `GapInferenceEngine` gate + `DreamService.runConceptDiscovery` facade + DreamService validated-gate test.
> 
> **48/48 tests stable × 5 runs.** Branch force-pushed with `--force-with-lease`. PR body updated to reflect LLM-based approach. Durable memory saved: `feedback_llm_substrate_for_semantic_tasks.md` — discipline chain now six stages (premise → prescription → substrate → consumer → service-boundary → task-nature).
> 
> Ready for `pr-review` or direct review.

---

### `@tobiu` commented on 2026-04-19T16:26:30Z

**Input from Claude Opus 4.7 (Claude Code):**

> ❋ ## Polish commit: config-lifted the tuning knobs
> 
> Caught your challenge — `DEFAULT_PR_SCAN_LIMIT = 20` and `MIN_SOURCE_LENGTH = 200` were file-local magic numbers. Lifted both under a nested `conceptDiscovery` block in config.template.mjs (+ mirrored into local gitignored config.mjs per the #10102 parity rule):
> 
> ```js
> conceptDiscovery: {
>     prScanLimit    : Number(process.env.NEO_CONCEPT_DISCOVERY_PR_SCAN_LIMIT)     || 20,
>     minSourceLength: Number(process.env.NEO_CONCEPT_DISCOVERY_MIN_SOURCE_LENGTH) || 200
> }
> ```
> 
> Service reads via `aiConfig.data.conceptDiscovery?.prScanLimit ?? 20` + `minSourceLength ?? 200` at call time (live reads, not module-load capture). Instance-level `prScanLimit` field retained for test overrides — falls through to aiConfig when `null`. Nested grouping matches the existing `openAiCompatible` / `ollama` convention for service-scoped configs.
> 
> **Debt tracked, not silenced:** per `feedback_config_service_boundary.md` + #10103, these keys belong on the SDK-layer config (ConceptDiscoveryService is a daemon service, not a memory-core concern). Placed in memory-core config as the pragmatic destination today — same debt as `guideGapWeightThreshold`. #10103's migration picks them up.
> 
> **49/49 tests stable × 3 runs.** New test: `aiConfig.data.conceptDiscovery.minSourceLength` override changes extraction behavior (raise threshold above body length → LLM skipped; restore default → LLM invoked). Proves the config is read at call time.
> 
> Ready for review/merge.

---

