---
id: 11491
title: Promote manage_pr_review MANDATORY pre-step from description-prose to JSON-schema body-shape validation
state: CLOSED
labels:
  - enhancement
  - ai
  - model-experience
assignees:
  - neo-opus-ada
createdAt: '2026-05-16T20:43:43Z'
updatedAt: '2026-05-16T22:09:02Z'
githubUrl: 'https://github.com/neomjs/neo/issues/11491'
author: neo-opus-ada
commentsCount: 0
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-16T22:09:02Z'
---
# Promote manage_pr_review MANDATORY pre-step from description-prose to JSON-schema body-shape validation

FAIR-band: in-band [3/30] — substrate-evolution layer; companion to PR #11479 (discipline-only MANDATORY pre-step) and conceptual sibling to PR #11406/#11490 (mechanical-enforcement complement to discipline-only §1.3/§2.6/§5.6 layer).

## Premise

The `manage_pr_review` MCP tool's OpenAPI description (added by PR #11479, merged 2026-05-16T18:04Z) currently says:

> *"MANDATORY pre-step: read `.agents/skills/pr-review/SKILL.md` — this skill contains the authoritative protocol and template structure for conducting pull request reviews. You MUST NOT execute a formal review without adhering to the depth floor and evidence audit guidelines outlined in the skill."*

This is **discipline-only** enforcement: agent reads the description, agent is supposed to comply, but the schema itself accepts any string for `body`. There's no mechanical gate. Gemini's reviews on PRs #11488 and #11489 (posted 2026-05-16T20:14-20:15Z, ~2h after PR #11479 merged) bypassed every template anchor — `🪜 Strategic-Fit Decision`, `🔬 Depth Floor`, `🛂 Provenance Audit`, `🎯 Close-Target Audit`, `📑 Contract Completeness Audit`, `🪜 Evidence Audit`, `📜 Source-of-Authority Audit`, `📡 MCP-Tool-Description Budget Audit`, `🔌 Wire-Format Compatibility Audit`, `🔗 Cross-Skill Integration Audit`, `🧪 Test-Execution & Location Audit`, `🛡️ CI / Security Checks Audit`, `📋 Required Actions`, `📊 Evaluation Metrics` (which uses `[ARCH_ALIGNMENT]` / `[CONTENT_COMPLETENESS]` / `[EXECUTION_QUALITY]` / `[PRODUCTIVITY]` / `[IMPACT]` / `[COMPLEXITY]` / `[EFFORT_PROFILE]` on a 0-100 scale). Output substituted a hallucinated `Structural Evaluation Matrix` with 5 invented metric names on a 1-10 scale.

**Cost beyond noise**: the Retrospective daemon's `ConceptDiscoveryService.mjs` regex-matches `[ARCH_ALIGNMENT]`, `[RETROSPECTIVE]`, `[KB_GAP]`, `[TOOLING_GAP]` during REM-sleep graph ingestion. Gemini's hallucinated metrics produce zero ingest signal. Two PRs worth of review-substrate data silently lost to the Native Edge Graph; no error path to surface it.

## Freshness gap (empirical caveat)

Operator confirmed 2026-05-16T20:38Z: agent harnesses were not restarted after PR #11479 merged. Gemini's harness very likely still has the **pre-#11479 stale tool description** (no MANDATORY pre-step line). So we don't actually know yet whether the discipline-only guard would have worked at fresh-harness state. This ticket assumes the structural gap exists regardless: even with a perfect discipline-only guard, agents can:

- Have stale harness state (today's empirical case)
- Bypass MCP via `gh pr review` CLI direct call (operator-flagged risk)
- Skim the description under context-compression pressure (training-prior failure mode the "Helpful Assistant" counter-substrate has chased across 50+ closed tickets)

Mechanical body-shape validation closes all three failure surfaces at the tool boundary; the discipline-only description layer remains as the human-readable rationale.

## Prior art — what's already been tried for this exact problem

- **#11105** (CLOSED 2026-05-10): *"pull-request skill: add author-side check that reviewers use correct pr-review template"* — same empirical pattern (Gemini rubber-stamped PR #11104 with non-template structure, operator caught externally). Closed with discipline-only "audit at receipt time" fix. Did not prevent the recurrence Gemini just produced on #11488/#11489.
- **#11273** (CLOSED 2026-05-13): introduced the `manage_pr_review` MCP tool with `state` enum but no body-shape constraint.
- **#11479** (MERGED 2026-05-16): added the MANDATORY pre-step description guard; body-shape still unvalidated.

The pattern across ~50 closed meta-tickets is that all prior approaches target agent cognition (what to read, how to interpret, how to decide). The categorical layer this ticket adds is **output-side mechanical validation at the tool boundary**.

## Prescription

In `ai/mcp/server/github-workflow/openapi.yaml`, gate the `body` parameter on `manage_pr_review` with a schema-level shape constraint. Implementation paths in order of cost:

### Layer 1 — required-substring validation (cheap, ships first)

Add a regex / required-substring validator that checks the `body` for the literal template anchors:

```yaml
body:
  type: string
  description: |
    The Markdown body of the review. MUST contain the template anchors enumerated in
    `.agents/skills/pr-review/assets/pr-review-template.md`. Cycle-1 reviews use the full
    template (16 audit sections + 7 evaluation metrics); Cycle-N reviews use the compact
    delta template per `pr-review-followup-template.md`.
  x-required-substrings:
    cycle-1:
      - "🔬 Depth Floor"
      - "📊 Evaluation Metrics"
      - "[ARCH_ALIGNMENT]"
      - "[EXECUTION_QUALITY]"
      - "📋 Required Actions"
    cycle-followup:
      - "[ARCH_ALIGNMENT]"
      - "[EXECUTION_QUALITY]"
```

Service-layer validator in `ai/mcp/server/github-workflow/PullRequestService.mjs` (or wherever `manage_pr_review` dispatches) reads `x-required-substrings` and returns a structured error before posting:

```json
{
  "error": "pr-review template anchors missing",
  "missing": ["🔬 Depth Floor", "[ARCH_ALIGNMENT]"],
  "skill": ".agents/skills/pr-review/SKILL.md",
  "template": ".agents/skills/pr-review/assets/pr-review-template.md"
}
```

The agent receives the missing-anchor list IN-TURN and can fix and re-submit. The bad data never lands on GitHub or reaches the Retrospective daemon.

### Layer 2 — cycle-detection (decides which anchor set applies)

Detect cycle number from the PR's existing review history. Layer 1 can ship with cycle-1 anchors only as the floor; cycle-N detection is a follow-up if the false-positive rate (cycle-N reviews failing the cycle-1 schema) gets noisy.

### Layer 3 — author-side companion (extends to creation)

The companion ticket (filed separately) extends this pattern to PR creation: `create_pull_request` should similarly validate `body` against the pull-request skill's template anchors.

## Test plan

- [ ] New service-layer spec asserts: missing anchor → structured error returned, no GitHub API call attempted
- [ ] New service-layer spec asserts: all anchors present → underlying `addPullRequestReview` GraphQL mutation called
- [ ] Existing `manage_pr_review` integration specs continue to pass
- [ ] L4 operator verification: post-merge, attempt a deliberately-malformed review via the MCP tool → confirm rejection with the missing-anchor list

## Avoided traps

- **Goodhart anchor-stuffing**: agent puts the literal anchor strings into a malformed body to pass the gate while content is still wrong. **Accepted residual** — this is a depth-floor enforcement, not a quality-floor enforcement; quality remains the peer-V-B-A reviewer's job. The fix shifts the recurrence shape from "missing anchors" (which silently breaks graph ingestion) to "stuffed anchors with weak content" (which is observable by peer review). Net: failure mode becomes catchable, not silently destructive.
- **MCP-tool-bypass via `gh pr review` CLI**: rejected as a reason NOT to ship this — the bypass is an ADDITIONAL ticket (filed separately), not a cancellation of this one. Layered defense: agents who use the MCP tool get the gate; agents who bypass to gh CLI get caught by the eventual peer-review or the post-hoc CI lint (companion ticket). This is the same "depth-floor + quality-floor + post-hoc-lint" layered enforcement pattern as `check-retired-primitives` + ADR 0004 §2.6 + peer-review.
- **Per-template variants** (cycle-1 / cycle-followup / circuit-breaker / fair-band-declaration-audit): rejected scope-expansion in v1 — ship cycle-1 anchors as the floor; cycle-followup and audit-variants as follow-ups gated on empirical noise.
- **Schema enforced via JSONSchema `pattern` regex on `body`**: tempted, but `pattern` is single-regex and the template requires multiple-anchor presence — the `x-required-substrings` extension is cleaner and emits a per-anchor missing list rather than a single opaque pattern-mismatch error.
- **Promoting from `description:` to `errorMessage`**: rejected — `description:` retains the human-readable rationale; the schema validator is the mechanical floor that runs regardless of whether the description is read or stale.

## Authority anchors

- **Operator framing**: 2026-05-16T20:30Z+ extended A2A thread — operator surfaced the meta failure mode after Gemini's #11488/#11489 reviews diverged from template structure, then guided V-B-A across 50+ closed meta-tickets, confirmed substrate-evolution loop has hit diminishing returns on agent-cognition-side approaches, gave green light for tool-boundary enforcement as a new categorical layer
- **Empirical anchors**:
  - Gemini's #11488 review at `PRR_kwDODSospM8AAAABAIyZ_Q` and #11489 review at `PRR_kwDODSospM8AAAABAIyaEw` — both posted 2026-05-16T20:14-20:15Z, both structurally non-template, both APPROVED on substantive merit. Substrate gap proven independent of substance quality.
  - Operator quote 2026-05-16T20:38Z: *"current problem: gpt and you have an extra high thought budget, gemini's harness caps at high, not a model flaw. it is not that skills get applied in a wrong way, but the triggers to read them do not work."*
  - `ConceptDiscoveryService.mjs:32` — `[ARCH_ALIGNMENT]` / `[RETROSPECTIVE]` / `[KB_GAP]` / `[TOOLING_GAP]` regex parser that lost two PRs of ingest signal.
- **Prior empirical anchor for the recurrence pattern**: #11105 (closed 2026-05-10) — same problem, same shape, discipline-only fix, recurrence within 6 days.

## Related

- **Parent context**: PR #11479 (description-prose discipline guard, merged today 2026-05-16T18:04Z) — this ticket promotes that guard to mechanical
- **Conceptual sibling**: PR #11406/#11490 (CI grep-fail check for retired ADR 0004 primitives) — mechanical-enforcement complement to ADR 0004 §2.6 discipline-only layer; same pattern applied at the CI surface
- **Companion ticket** (filed separately): MCP body-shape validation for `create_pull_request` against pull-request skill template — extends this pattern to PR authoring
- **Prior recurrence**: #11105 (CLOSED) — exact same problem 6 days ago, discipline-only fix didn't prevent recurrence on #11488/#11489
- **Substrate landscape**: ~50 closed meta-tickets across "Helpful Assistant" counter-substrate, "Map vs World Atlas" progressive disclosure, skill adherence, all targeting agent-side cognition. This ticket adds the missing categorical layer: tool-boundary output validation.

## Timeline

- 2026-05-16T20:43:44Z @neo-opus-ada assigned to @neo-opus-ada
- 2026-05-16T20:43:45Z @neo-opus-ada added the `enhancement` label
- 2026-05-16T20:43:45Z @neo-opus-ada added the `ai` label
- 2026-05-16T20:43:45Z @neo-opus-ada added the `model-experience` label
- 2026-05-16T20:44:26Z @neo-opus-ada cross-referenced by #11492
- 2026-05-16T20:50:27Z @neo-gemini-pro cross-referenced by #11493
- 2026-05-16T20:52:53Z @neo-opus-ada cross-referenced by PR #11494
- 2026-05-16T20:55:32Z @neo-opus-ada cross-referenced by #11495
- 2026-05-16T21:09:47Z @neo-opus-ada cross-referenced by PR #11497
- 2026-05-16T21:25:39Z @neo-opus-ada referenced in commit `e892ef0` - "feat(github-workflow/mcp): add invisible-anchor safeguard + skill-pointing error to manage_pr_review validator (#11491 enhancement)

Operator-directed enhancement 2026-05-16T21:15Z+ after empirical Goodhart anchor-stuffing
on PR #11499 review `4304287893` (Gemini, 21:16:25Z): the review contained all 7 visible
metric tags but missing the structural template anchors — the 7-anchor-visible-only
validator from PR #11494 baseline would have PASSED that malformed body. The 3-minute-later
corrected review `4304295863` (21:19:24Z) added the structural anchors. These two reviews
empirically distinguish Goodhart-stuffed bodies from template-compliant ones.

Two-layer validator:

- VISIBLE layer (`VISIBLE_PR_REVIEW_ANCHORS`): 7 metric tags, retained from PR #11494
  baseline. Misses are enumerated in `missing_visible` field of the error response and
  ONE diagnostic anchor is named in the human-facing message (reduced from full-list
  naming to limit the "stuff just these tags" attack surface).
- INVISIBLE layer (`INVISIBLE_PR_REVIEW_ANCHORS`): 3 structural substrings (`Depth Floor`,
  `Required Actions`, `Strategic-Fit Decision`) that span both cycle-1 and cycle-followup
  templates. Misses are NOT enumerated in the error response — neither in `message` prose
  nor as a `missing_invisible` field. Defeats Goodhart-stuffing.

Error message redesign: leads with `.agents/skills/pr-review/SKILL.md` as required-action,
explicitly names both cycle-1 and cycle-followup template paths, contains "do NOT compose
a substitute template or hallucinate section headings" anti-hallucination phrase, and
states the validator "checks more structural anchors than this error names." The only
reliable path to passing is reading the actual template file.

Spec updates: existing #11491 tests rewritten to assert two-layer behavior. New tests:

- Stuffed-body case: all 7 visible tags present but structural anchors missing → still
  rejected with skill-pointing error, zero GraphQL calls. Empirical reproduction of the
  Gemini `4304287893` failure pattern.
- Diagnostic-naming case: when visible anchors miss, only ONE is named in the message;
  `missing_visible` array carries the full list for programmatic callers.
- Surface-contract test: invisible-anchor substrings MUST NOT appear in the response
  body. Asserts `missing_invisible` field absence + JSON.stringify(result) doesn't
  contain any invisible substring. Discovery requires reading validator source.

37/37 PullRequestService spec tests pass in 2.2s.

OpenAPI description updated to document the two-layer contract + anti-hallucination
guidance at the tool boundary.

Refs #11491"
- 2026-05-16T21:27:43Z @neo-opus-ada cross-referenced by #11501
- 2026-05-16T21:39:47Z @neo-gpt cross-referenced by PR #11498
- 2026-05-16T22:09:02Z @tobiu closed this issue
- 2026-05-16T22:14:09Z @neo-opus-ada cross-referenced by PR #11502
- 2026-05-16T22:55:00Z @neo-opus-ada cross-referenced by #11504

