# Stage 3 — Report reference

Covers confidence scoring, REVIEW.md assembly, pre-publish inspection gates, and the publish paths (PR mode + local mode).

---

## Confidence scoring (per case)

After a case completes, you (the agent) assign a single confidence score 0–100. The aggregator does not compute this — you have the full context (STEP_PASS counts, evidence types, iteration count, console state, screenshot quality) and are in the best position to judge.

### Guidance ranges

| Range | When to use |
|---|---|
| 95–100 | All steps pass at iter 1. Evidence is `deterministic` or `a11y_ref` (no soft judgment). Console clean. Screenshot matches expectation cleanly. |
| 85–94 | All steps pass at iter 1 with minor benign variations (whitespace, ordering, capitalization). OR pass after one iter-2 adjustment that doesn't change meaning (selector typo etc.). |
| 75–84 | Pass after iter-2 adjustment. OR iter-1 pass where most evidence is `screenshot_judgment` rather than deterministic. |
| 60–74 | Pass after iter-3 with full instrumentation. OR iter-1 pass where the critical step had weaker evidence than ideal. |
| 40–59 | Critical step failed but you're not sure whether the bug is in the diff or in the test setup. Surface for human review. |
| 0–39 | Confirmed failure — feature doesn't work as the description claims. |

### Inputs to consider

1. **How many steps passed** vs. failed
2. **Which steps failed** — critical AC-verifying step, or peripheral check?
3. **Evidence type quality** — `deterministic`/`a11y_ref` vs `screenshot_judgment`
4. **Iteration count** — iter-1 strongest, iter-3 shakiest
5. **Health-gate state** — did the page break at all during the case?
6. **Console state** — errors that didn't break things but suggest fragility?

### Aggregator behavior

`aggregate_verdicts.py` reads your per-case confidence and applies a deterministic mapping:

| Condition | Label |
|---|---|
| All cases confidence ≥ 75 AND no flaky cases | `ai-approved` |
| All cases confidence ≥ 60 AND ≤ 2 flaky cases | `ai-needs-human` |
| Any case confidence < 60 OR ≥ 3 flaky cases | `ai-rejected` |
| No cases executed (infra-only / empty plan) | `ai-needs-human` |

No penalty math overlaid on your score. Your number stands.

### Anti-patterns

- ❌ **Splitting the difference**: scoring 70 because "I'm not sure" is a cop-out. Dig deeper (iter 2/3) or score honestly into 40–59.
- ❌ **Inflating to avoid `ai-rejected`**: if the feature is broken, give it a low confidence. Don't game the label.
- ❌ **Scoring 100 ever**: cap at 99 unless you have written deterministic proof of every claim.
- ❌ **Scoring without reading your own STEP_PASS lines**: the line count and evidence types are inputs, not afterthoughts.

### Rationale required

In the per-case verdict JSON, include a 1–3 sentence `rationale` field explaining the score.

---

## REVIEW.md assembly

The aggregator writes `verdict-final.json`; you assemble `REVIEW.md` from it plus your case sections. Required sections, in order:

1. **Title + idempotency footer** at the very top:
   ```markdown
   # Business Review — <PR title or commit subject> — `br-sha:<SHORT_SHA>`
   ```
2. **Requirements summary** — your own words, ≤ 5 lines per AC + one overview paragraph. Reference AC by ID.
3. **`### Case <id> — <name>` sections** — one per case, including `**Confidence:**` line + rationale + key evidence (file refs into `case<N>/`).
4. **Summary** — verdict label, avg_confidence, case_summary table (from `verdict-final.json`).
5. **Coverage** — `change_types` touched, `iter_distribution` (iter-1/2/3 counts), `evidence_summary` (counts by evidence type), `health_gate_failures` count.
6. **AC coverage matrix** — table per AC → covering cases → pass/fail.
7. **Findings to act on** (see below).
8. **Blockers** (filled or "None").
9. **Deployment notes** (filled or "None").
10. **Limitations** (explicit gaps; in local mode include any dirty-tree paths if autostash ref active).
11. **Local artifacts footer** (paths on this machine).
12. Closing line: `*Generated by AI business review — br-sha:<SHORT_SHA>*`

REVIEW.md uses local relative paths for screenshots. No image-URL substitution downstream.

### Findings to act on (downstream-AI handoff contract)

The **Findings to act on** section is the contract for the next AI session that picks up the work. The structure must be:

```markdown
## Findings to act on

### 1. case-3.1 — Required field allows whitespace [REJECTED, confidence 35]

**What failed:** Submit with `name: "   "` succeeded; expected validation error.

**Where:** `app/Filament/Resources/CompanyResource.php:142` — `name` field declaration is missing `->rule('trim:min:1')` or similar.

**Reproduce:**
```
agent-browser open "$RELATICLE_URL/app/<team>/companies/create"
agent-browser fill 'input[name="data.name"]' "   "
agent-browser click 'button[type="submit"]'
# Result: redirect to /companies/<id>; expected validation error
```

**Suggested next action:** Add a `Rule::trim()->minLength(1)` on the name field; write a Pest test in `tests/Feature/Filament/CompanyResourceTest.php` covering the whitespace case.
```

Each finding has: severity tag, what failed, file:line ref, copy-pasteable repro, suggested next action.

**Order findings by severity:** rejected cases first, then needs-human, then approved-but-flagged. Omit purely-passing cases — they don't need follow-up.

In PR mode this section is still useful as a reviewer summary; the audience is the human PR author rather than a downstream AI.

---

## Publishing gates (6b/6c)

Both gates run before any GitHub write OR local-mode finalization.

### Gate 6b: REVIEW.md file integrity

Read REVIEW.md end-to-end and verify:

1. Every case section has a `**Confidence:**` line with a 0-100 integer. Count must equal the number of `### Case` sections.
2. Summary counts match reality. Count cases at each confidence band (≥75, 60-74, <60); confirm Summary totals match.
3. The requirements summary is in agent's own words, not copy-pasted from the PR description.
4. No placeholder text remains: grep for `<placeholder>`, `TBD`, `<N>`, `<one or two sentences>`. Every placeholder must be resolved.
5. No `/tmp/` paths leaked into the body.
6. Blockers / Deployment notes / Limitations sections are filled in or explicitly say "None".
7. Coverage section present with `change_types`, `iter_distribution`, `evidence_summary`, `health_gate_failures` count.

### Gate 6c: PNG sanity

For each `case<N>/screenshot.png`:

1. Visible red callout box drawn around the element the case claims to prove.
2. Literal evidence text legible inside the box at native resolution.
3. No critical content clipped at any edge.
4. No stale callouts from previous cases leaked in.
5. Filename matches the case it claims to show.

If either gate fails, fix or retake before proceeding. No exceptions.

**Local-mode exception:** Gate 6c is skipped only if zero cases took screenshots (e.g. all `pest-only` cases or all browser cases used `screenshot: {none: ...}`). Otherwise it runs as in PR mode.

---

## Publish paths

### Push decision matrix

| Invocation | Behavior |
|---|---|
| `--publish` (PR mode only) | Runs `publish.sh` immediately. No prompt. Prints comment URL + applied label. |
| `--no-prompt` | Writes `publish.sh` (PR mode) or skips it (local mode). Prints path to REVIEW.md. **No GitHub writes.** |
| (default, PR mode) | Writes `publish.sh`, then prints structured end-of-run prompt (see below). Runs `publish.sh` on `y`. |
| (default, local mode) | Skips `publish.sh`. Prints structured prompt offering to push if user supplies a PR number. |

### `publish.sh` template (PR mode)

```bash
#!/bin/bash
set -e

LABEL=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["label"])' \
        "$REVIEW_DIR/verdict-final.json")

gh pr comment "$PR_NUM" --repo "$REPO" -F "$REVIEW_DIR/REVIEW.md"

# Always remove other ai-* labels before adding the new one
for OLD in ai-approved ai-rejected ai-needs-human; do
  gh pr edit "$PR_NUM" --repo "$REPO" --remove-label "$OLD" 2>/dev/null || true
done
gh pr edit "$PR_NUM" --repo "$REPO" --add-label "$LABEL"

# Write the posted comment ID for one-off retract
gh pr view "$PR_NUM" --repo "$REPO" --json comments \
  --jq '.comments[-1].id' > "$REVIEW_DIR/posted-comment-id.txt"

echo "Posted $LABEL to PR #$PR_NUM"
```

### Local-mode finalization

After gates pass:

```bash
echo "$REVIEW_DIR  $(jq -r .label "$REVIEW_DIR/verdict-final.json")" \
  > .context/reviews/local/LATEST.txt
```

`.context/reviews/local/LATEST.txt` is the well-known pointer the next AI session reads to find the most recent review without scanning directories.

### Label management

`ai-approved` / `ai-rejected` / `ai-needs-human` labels must exist on the repo. Create once:

```bash
gh label create ai-approved --repo "$REPO" --color 0e8a16 --description "AI business review: approved" 2>/dev/null || true
gh label create ai-rejected --repo "$REPO" --color b60205 --description "AI business review: rejected" 2>/dev/null || true
gh label create ai-needs-human --repo "$REPO" --color fbca04 --description "AI business review: needs human verdict" 2>/dev/null || true
```

Idempotent — silently no-ops if labels already exist.

### Idempotency marker

Every PR comment must end with `*Generated by AI business review — br-sha:<SHORT_SHA>*`. Stage 1 scans for this marker to detect "already reviewed at this SHA" and skip.

---

## End-of-run prompt (structured)

The end-of-run prompt is NOT a shell `read -p` (this skill runs inside Claude Code; stdin may be unavailable). Instead, print a structured prompt block and yield control to the parent agent.

Print this block at the end of Stage 3 when `PUBLISH_AUTO=0` AND `NO_PROMPT=0`:

```
========== BUSINESS REVIEW COMPLETE ==========
Mode: <pr | local>
Verdict: <verdict-final.json .label>
Avg confidence: <verdict-final.json .avg_confidence>
Cases: <verdict-final.json .case_count> (<#flaky> flaky)
Findings: <count> items need attention
Report: <REVIEW_DIR>/REVIEW.md

Push report? [y/N]
- 'y' (PR mode): runs <REVIEW_DIR>/publish.sh
- 'y' (local mode): asks for PR number, then runs publish.sh
- 'N' or anything else: exits without posting
==============================================
```

After printing, exit the skill cleanly. The parent agent collects the user's response and, on `y`, invokes the skill again with `--publish` (PR mode) or runs `publish.sh` directly with the supplied PR number (local mode).

For `PUBLISH_AUTO=1`: run `bash $REVIEW_DIR/publish.sh` immediately; exit.
For `NO_PROMPT=1`: print only `"Report: $REVIEW_DIR/REVIEW.md"`; exit.

In ALL cases, 6b + 6c gates run BEFORE the prompt or publish — if either gate fails, print the failure and exit non-zero without prompting or publishing.

---

## Hard rules

- Both gates (6b and 6c, where applicable) must pass before any publish or finalization. No exceptions.
- Never post a REVIEW.md that contains placeholder text, `/tmp/` paths, or unresolved `<N>` tokens.
- Always remove the old `ai-*` label before adding the new one — never stack two `ai-*` labels.
- The idempotency marker (`br-sha:<SHORT_SHA>`) must appear verbatim at the end of every posted comment.
- On `--publish`, run `publish.sh` directly — never prompt.
- On default (no flag), print the structured prompt and exit; do not block on stdin.
- Never publish to GitHub from pure local mode without an explicit PR number supplied at the prompt.
- The aggregator NEVER overrides agent-assigned confidence. Don't tweak `verdict-final.json` after the fact.
