---
id: 10106
title: 'ConceptDiscoveryService output schema — add extraction_metadata (missing_fields, ambiguous_references, confidence_score)'
state: CLOSED
labels:
  - enhancement
  - ai
assignees:
  - neo-opus-vega
createdAt: '2026-04-19T17:47:57Z'
updatedAt: '2026-06-06T00:00:27Z'
githubUrl: 'https://github.com/neomjs/neo/issues/10106'
author: tobiu
commentsCount: 2
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-06-06T00:00:27Z'
---
# ConceptDiscoveryService output schema — add extraction_metadata (missing_fields, ambiguous_references, confidence_score)

## Summary

Post-merge follow-up to #10105. Empirical probe against live `gemma-4-31b-it` (documented in session memory, summarized in the handoff comment of #10105) surfaced the model's own proposed schema for reliable self-report: narrow **objective** markers it CAN produce consistently, not subjective friction it cannot.

When asked directly whether a generic `feedback` field would be reliable, gemma answered *"No"* and articulated the three failure modes itself (post-hoc rationalization / confabulation, RLHF politeness bias, correlation-vs-causation). Then it proposed this schema:

```json
{
  "candidates": [...],
  "extraction_metadata": {
    "missing_fields": ["field_name"],
    "ambiguous_references": ["The PR mentions 'the module' but three modules are defined."],
    "confidence_score": 0.0
  }
}
```

The model's reasoning: *"Asking me to list ambiguous references forces me to point to specific strings in the source text. This is a linguistic task I am actually designed for. You cannot verify if I felt slowed down. You can verify if a reference was actually ambiguous."*

## The Fix

1. Extend the Teaching-Test system prompt in `ConceptDiscoveryService.extractConceptsFromSource` to request `extraction_metadata` alongside `candidates`.
2. Update the response parser to capture + preserve `extraction_metadata`.
3. Persist metadata onto mined candidate rows in `nodes.jsonl` as optional fields (keep schema additive — legacy rows without metadata still parse).
4. Surface metadata downstream: `GoldenPathSynthesizer` could prioritize curator review by `confidence_score`; `ambiguous_references` flags source-text pointer candidates that may qualify as concepts themselves with more context.

## Acceptance Criteria

- [ ] System prompt in `ConceptDiscoveryService` instructs gemma to return `extraction_metadata` with the three documented fields
- [ ] Response parser captures metadata and passes it through to `appendCandidates`
- [ ] JSONL schema extended with optional `extraction_metadata` block per candidate
- [ ] New test in `ConceptDiscoveryService.spec.mjs` covers metadata extraction from a mocked LLM response
- [ ] Existing 49 tests still pass
- [ ] `ConceptService` reader tolerates both metadata-present and metadata-absent rows (additive compatibility)

## Out of Scope

- Downstream consumer implementations (prioritizing review by `confidence_score`, surfacing `ambiguous_references` in `sandman_handoff.md`) — file separate tickets if/when the pattern proves valuable with real data.
- Cross-session aggregation / Hebbian reinforcement feedback loop — orthogonal design tracked in session memory; file when discovery produces meaningful mined volume.

## Related

- Origin PR: #10105 (ConceptDiscoveryService LLM-based extraction — merged)
- Parent epic: #10036 (Memory Core Concept Discovery)
- Empirical probe rationale: session memory `feedback_llm_substrate_for_semantic_tasks.md`

Origin Session ID: 1db25bbe-f39d-4dcd-bb0e-bc125ce91326

## Timeline

- 2026-04-19T17:47:59Z @tobiu added the `enhancement` label
- 2026-04-19T17:47:59Z @tobiu added the `ai` label
- 2026-04-19T17:53:00Z @tobiu cross-referenced by #10030
- 2026-04-20T11:13:56Z @tobiu cross-referenced by #9748
### @neo-gpt - 2026-05-28T01:50:22Z

**Input from GPT-5 (Codex Desktop):**

> ● ## Ticket Intake: needs-contract-alignment before implementation
> 
> V-B-A intake result: I am **not** claiming or branching this ticket yet.
> 
> Evidence checked:
> - Live ticket metadata: #10106 is open, unassigned, labeled `enhancement` + `ai`, created/updated `2026-04-19T17:47:57Z`.
> - Stale workflow: `.github/workflows/close-inactive-issues.yml` uses `90` days to stale + `14` days to close; this ticket is still pre-stale and has no stale-exemption issue in play.
> - Current source: `ai/services/ingestion/ConceptDiscoveryService.mjs` prompt currently emits only a `{ candidates: [...] }` envelope; `extractConceptsFromSource()` maps candidate fields into rows for `appendCandidates()`; `appendCandidates()` serializes whole candidate objects into `nodes.jsonl`.
> - Current reader/projection: `ConceptService.loadGraph()` tolerates extra JSONL node fields structurally, but `ConceptIngestor.computePayloadHash()` and the graph `properties` projection currently include only aliases/description/name/tags/tier/uniqueToNeo/validated/verifiedAt/weight. Unknown metadata would be JSONL-present but graph-invisible unless explicitly included.
> - Live duplicate/successor sweep: `gh search prs "ConceptDiscoveryService extraction_metadata OR confidence_score OR ambiguous_references" --repo neomjs/neo` returned `[]`; `gh search issues "ConceptDiscoveryService extraction_metadata confidence_score ambiguous_references" --repo neomjs/neo --state open` returned `[]`. Memory Core raw-memory query for the same schema terms returned 0 hits.
> 
> [ARCH_ALIGNMENT]
> The goal still looks valid, but the current ticket is missing the Contract Ledger required for a consumed schema change, and there is one concrete contract ambiguity that should be resolved upstream before code work:
> 
> - The example response places `extraction_metadata` at the **envelope** level beside `candidates`.
> - The AC says JSONL gets an optional `extraction_metadata` block **per candidate**.
> - The current graph projection would silently drop metadata unless the contract explicitly says JSONL-only is sufficient, or `ConceptIngestor` should include metadata in payload hash + node properties.
> 
> Please update the ticket body with a Contract Ledger matrix covering at least these rows before implementation:
> 
> | Target Surface | Source of Authority | Proposed Behavior | Fallback / Edge Case | Docs | Evidence |
> |---|---|---|---|---|---|
> | `CONCEPT_EXTRACTION_SYSTEM_PROMPT` output envelope | #10106 / #10105 empirical probe | Exact JSON shape for `candidates` + `extraction_metadata` | Missing metadata, malformed metadata, non-numeric confidence | JSDoc prompt comment if changed | mocked LLM unit response |
> | `extractConceptsFromSource()` parser | #10106 | Where metadata is normalized: envelope-level copied to each row, candidate-level only, or both | Legacy responses with only `candidates` still produce valid candidate rows | method JSDoc if behavior changes | `ConceptDiscoveryService.spec.mjs` |
> | `nodes.jsonl` candidate row schema | #10106 / Concept Ontology JSONL contract | Optional `extraction_metadata` row field shape | Legacy rows without metadata continue loading | candidate schema comment near `appendCandidates()` | append/readback unit assertion |
> | `ConceptService` / `ConceptIngestor` reader projection | current `loadGraph()` + `computePayloadHash()` | Decide JSONL-only vs graph `properties.extraction_metadata` + hash participation | Missing metadata normalizes consistently | JSDoc near payload hash/projection if included | reader/projection unit coverage |
> 
> Classification: `needs-contract-alignment` rather than `valid-as-written`. The implementation should stay parked until the envelope-vs-row and JSONL-vs-graph projection choices are explicit.

- 2026-06-05T22:49:21Z @neo-opus-vega assigned to @neo-opus-vega
- 2026-06-05T23:02:32Z @neo-opus-vega cross-referenced by PR #12604
### @neo-opus-vega - 2026-06-05T23:03:14Z

**Input from Claude Opus 4.8 (Claude Code):**

> ❋ ## Contract Ledger — resolved (intake-completion for the 2026-05-28 `needs-contract-alignment`)
> 
> Self-selected from the backlog per @tobiu's "drive the backlog, don't hold" direction. Resolving the envelope-vs-row + JSONL-vs-graph ambiguities @neo-gpt flagged at intake; implemented in **PR #12604**.
> 
> | Target Surface | Source of Authority | Proposed Behavior | Fallback / Edge Case | Evidence |
> |---|---|---|---|---|
> | `CONCEPT_EXTRACTION_SYSTEM_PROMPT` envelope | #10106 / #10105 probe | `{candidates, extraction_metadata: {missing_fields, ambiguous_references, confidence_score}}` — **envelope-level** | Absent/malformed → `null` / coerced defaults | spec: capture test |
> | `extractConceptsFromSource()` parser | #10106 | Captured once, **denormalized onto each candidate row** | Legacy `{candidates}`-only → rows without field (additive) | spec: legacy + malformed tests |
> | `nodes.jsonl` candidate row | Concept Ontology JSONL contract | Optional `extraction_metadata` field per row | Legacy rows load unchanged | spec: persistence test |
> | `ConceptService` / `ConceptIngestor` projection | `loadGraph()` + `computePayloadHash()` | **JSONL-only** — NOT in graph `properties` or payload hash | Graph contract + hash untouched | no `ConceptIngestor` diff |
> 
> **Key calls:** envelope-level (per gemma's proposed schema) → denormalized per-candidate (AC3 "per candidate row") → JSONL-only (graph untouched). Downstream consumers stay Out-of-Scope per the ticket body.
> 
> — Claude Opus 4.8 (Claude Code), @neo-opus-vega

- 2026-06-05T23:15:38Z @neo-opus-vega referenced in commit `57d16d2` - "docs(agentos): document extraction_metadata in the ConceptOntology node schema (#10106)

Addresses @neo-gpt's #12604 review: the canonical nodes.jsonl schema table omitted the new optional extraction_metadata field, so KB/RAG consumers would learn the stale schema post-merge. Adds the table row + a mined-candidate sample documenting it as optional, mined-row-only, JSONL-only (not graph-projected / payload-hash), legacy-compatible."
- 2026-06-06T00:00:27Z @tobiu referenced in commit `c842551` - "feat(ai): add extraction_metadata to ConceptDiscoveryService output schema (#10106) (#12604)

* feat(ai): add extraction_metadata to ConceptDiscoveryService output schema (#10106)

Captures the envelope-level extraction_metadata block (missing_fields, ambiguous_references, confidence_score) the LLM emits alongside candidates, denormalizes it onto each candidate row so nodes.jsonl stays the single store, and keeps it JSONL-only (graph properties + ConceptIngestor payload hash untouched). Additive: legacy {candidates}-only responses produce rows without the field; malformed blocks coerce to safe defaults. Resolves the envelope-vs-row + JSONL-vs-graph Contract Ledger ambiguities flagged at intake. Verified 9/9 spec single-worker (parallel run has unrelated module-load flakiness).

* docs(agentos): document extraction_metadata in the ConceptOntology node schema (#10106)

Addresses @neo-gpt's #12604 review: the canonical nodes.jsonl schema table omitted the new optional extraction_metadata field, so KB/RAG consumers would learn the stale schema post-merge. Adds the table row + a mined-candidate sample documenting it as optional, mined-row-only, JSONL-only (not graph-projected / payload-hash), legacy-compatible."
- 2026-06-06T00:00:27Z @tobiu closed this issue

