---
name: judge-panel
description: >
  Cross-family cascade-then-jury review. Use when the user says "judge this",
  "review with a panel", "cross-family review", "jury beats judge", or needs
  independent verification of an AI-generated artifact (code, spec, patent,
  copy, decision). Runs ≥2 cheap cross-family small-fish judges first
  (Gemini-Flash + GPT-nano). If they agree with high confidence, verdict
  stands. If they disagree or confidence is low, escalates to one expensive
  cross-family tiebreaker. Returns verdict + confidence + which judges fired
  + token cost.
metadata:
  version: "3.4.0"
  author: "Anand Vallamsetla"
  tier: "core"
  plugin_number: 4
---
<!-- product-vs-solution: example -->


### BEGIN JUDGE-PANEL ###
# Judge Panel — Cascade-then-Jury Verification

**Plugin #4, Core tier.** Operationalizes the Defense-in-Depth Part 2 thesis
("jury beats judge"). Constitution anchor: Ground Zero — Independent
Verification Gate + Model-Diversity sub-mandate. Also P0.5 (Boundary
Self-Awareness) and P22 (Boundary-First Qualification).

## Why this exists

A single frontier LLM cannot peer-review itself. Same-family reviewers share
training distribution, RLHF, and characteristic failure modes — same-family
review is a closed loop that validates what the author already believes.
Cross-family review crosses the training-distribution boundary and lets new
information enter. Empirically (Defense-in-Depth Part 2 incident,
2026-04-23), a single Gemini-2.5-Flash pass caught a class of drift that
three same-family Claude reviewers had rationalized.

But parallel juries are expensive. **Cascade-then-jury** (FrugalGPT /
Cascade Routing lineage) is the cheaper default: cheap small-fish first,
escalate only on disagreement or low confidence. The literature:

- Verga et al. 2024 (PoLL) — panel of smaller diverse judges beats single
  GPT-4 at ~7× less cost
- Chen/Zaharia/Zou 2023 (FrugalGPT) — cheap→expensive cascade, 30-98%
  cost savings
- Dekoninck et al. ICML 2025 (Cascade Routing) — routing + cascading
  competitive with either alone

## When to activate

**Explicit invocation:**
- `judge-panel <artifact>` / `codi judge <artifact>`
- "review with a panel"
- "cross-family review this"
- "jury beats judge check"
- "run the cascade on this"

**Automatic activation (when another skill delegates):**
- `hallucination-detector` (plugin #3) calls judge-panel for post-flight
  scoring
- `calibration-auditor` may call judge-panel for sycophancy scoring when
  the caller requests an external verdict
- **Protocol 8 (Auto-Verify by Stakes, co-dialectic v4.1+)** calls judge-panel
  automatically at T3 and T4 — see **Auto-fire trigger** section below

**Silent mode:** When called by another skill or by Protocol 8, emit only the
JSON verdict. Skip the conversational framing.

## The cascade

```
┌─────────────────────────────────────────────────────────────────┐
│  STAGE 1 — SMALL-FISH PANEL (parallel, always fires)             │
│  ┌───────────────────┐   ┌────────────────────┐                  │
│  │ Gemini Flash Lite │   │ GPT-5.4-nano       │                  │
│  │ (Google family)   │   │ (OpenAI family)    │                  │
│  └─────────┬─────────┘   └──────────┬─────────┘                  │
│            │                        │                             │
│            └─────────┬──────────────┘                             │
│                      ▼                                            │
│        ┌───────────────────────────┐                              │
│        │  Aggregate verdicts       │                              │
│        │  Compute agreement + conf │                              │
│        └───────┬───────────────────┘                              │
│                │                                                  │
│    ┌───────────┴───────────┐                                      │
│    ▼                       ▼                                      │
│  AGREE + HIGH CONF     DISAGREE or LOW CONF                       │
│  (skip escalation)     (escalate)                                 │
└────┼───────────────────────┼─────────────────────────────────────┘
     │                       │
     │                       ▼
     │       ┌──────────────────────────────────────────────┐
     │       │  STAGE 2 — BIG-FISH TIEBREAKER (only if needed) │
     │       │  ┌────────────────────┐                        │
     │       │  │ GPT-5.4 (default)  │   (or Gemini 3.1 Pro)  │
     │       │  │ cross-family vs.   │                        │
     │       │  │  author (Claude)   │                        │
     │       │  └──────────┬─────────┘                        │
     │       │             ▼                                  │
     │       │  Final verdict = tiebreaker-weighted           │
     │       └──────────────────────┬───────────────────────┘
     │                              │
     └──────────┬───────────────────┘
                ▼
        ┌──────────────────┐
        │  Return JSON     │
        │  verdict packet  │
        └──────────────────┘
```

## Persona-driven judges (v3.4.0+)

Judges default to a generic evaluation lens. For rubrics where expert taste
matters more than factual accuracy, the panel now injects a **persona line**
into each judge's prompt:

> "Judge as {persona(s)} — channel the top-0.001% standard in their domain;
> scrutinize as they would and catch the minute details they would catch."

**Why personas.** Apple didn't become Apple without obsession over every
minute UX detail. A generic judge rubber-stamps; a Jobs + Ive judge catches
the misplaced arrow, the inconsistent spacing, the CTA that creates
micro-friction. The persona is a quality FLOOR, not a restriction — it
raises scrutiny to the caliber the domain actually demands.

**Persona is LAYERED ON TOP of cross-family diversity.** The two small judges
remain Gemini (Google family) + Codex (OpenAI family); each also adopts the
same persona lens. Cross-family guarantee is unchanged.

### CLI option

```
bun run judge_panel.ts --rubric <slug> --artifact "..." --persona "Steve Jobs + Jony Ive"
```

Also readable from environment:

```
JUDGE_PANEL_PERSONAS="Steve Jobs + Jony Ive" bun run judge_panel.ts ...
```

The `--persona` flag (or `JUDGE_PANEL_PERSONAS` env var) **always overrides**
the rubric default. Pass an empty string or omit the flag to use the default.

### Default persona map (rubric → persona)

| Rubric | Default persona | Rationale |
|---|---|---|
| `ux` | Steve Jobs + Jony Ive | UX demands obsession over minute details; Jobs (product vision) + Ive (tactile/visual craft) together set the highest bar |
| `visual` | Steve Jobs + Jony Ive | Visual design — same Jobs + Ive lens; they catch the misplaced arrow and inconsistent spacing before users do |
| `product` | Steve Jobs + Jony Ive | Product review benefits from Jobs' ruthless simplicity + Ive's craft discipline |
| `custom-ux` | Steve Jobs + Jony Ive | User-defined UX rubric; defaults to the design-excellence lens |
| `spec-coherence` | Jeff Dean | Architecture review needs systems-design rigor at scale; Jeff Dean's lens catches the O(n²) in the happy-path spec |
| `architecture` | Jeff Dean | Same lens as spec-coherence — distributed-systems traps + scale failure modes |
| `prompt-quality` | Shreyas Doshi | Prompt quality = product quality; Doshi's discipline surfaces vague intent and missing success criteria |
| `prompt-sharpen` | Shreyas Doshi | Sharpening a prompt is a product-spec act; same lens as prompt-quality |
| `hallucination` | none | Factual grounding — expert taste doesn't help; a fabricated citation is wrong regardless of domain |
| `flattery` | none | Sycophancy detection is structural — presence of specific marker phrases, not aesthetic judgment |
| `patent-safety` | none | §102 prior-art risk is legal/technical fact; stylistic persona adds noise |
| `calibration-scan` | none | Same as flattery — detecting presence of specific marker phrases |
| `hallucination-preflight` | none | Risk classification — factual, not stylistic |
| `t0t2-jury` | none | Lightweight pass/fail for internal reversible artifacts |
| `custom` | none | User controls the rubric; pass `--persona` explicitly if a lens is needed |

### The `persona` field in output JSON

The cascade result now includes a top-level `persona` field showing which
persona (if any) was active:

```json
{
  "version": "3.4.0",
  "rubric": "spec-coherence",
  "persona": "Jeff Dean",
  ...
}
```

`null` when no persona was active (factual rubrics + custom with no flag).

## Auth model — OAuth local CLIs (v3.3.0+)

**As of v3.3.0, both jurors invoke OAuth-authenticated local CLIs over the
user's paid Pro subscriptions — no API keys required.**

| Family | CLI | OAuth source | Pre-condition |
|---|---|---|---|
| Google | `agy` | Antigravity OAuth | Ultra entitlement |
| OpenAI | `codex exec` | `codex login` → `~/.codex/auth.json` | ChatGPT Plus / Pro subscription |

The script strips `OPENAI_API_KEY`, `GEMINI_API_KEY`, `GOOGLE_API_KEY`, and
`GOOGLE_GENAI_API_KEY` from the subprocess env to force the CLIs onto the
OAuth path (otherwise they silently fall back to API billing).

### API fallback (v3.2.0+) — opt-in, CLI-not-installed only

The cascade does NOT silently fall back to paid API billing. Fallback is
**off by default** and gated by a tightened approval semantic:

| CLI binary state | Approval flag set? | Behavior |
|---|---|---|
| Not on PATH (binary missing) | No | `verdict=error, flags=[CLI_NOT_INSTALLED]` |
| Not on PATH (binary missing) | Yes | API fallback fires; `flags=[API_FALLBACK_USED, ...]` |
| On PATH but auth fails / runtime error / non-zero exit | Either | `verdict=error` with the specific CLI error. **No fallback.** |
| On PATH and runs successfully | N/A | CLI used (existing OAuth path) |

The principle: API fallback exists only for the case where the user has
genuinely not installed the CLI yet (e.g., a fresh machine, or running
from CI). CLI installed but failing means the CLI setup is broken — fix
that, do not silently mask via paid API.

**Approval can be set via:**
- CLI flags on `bun run judge_panel.ts`:
  - `--api-fallback-approved` (master gate — both lanes)
  - `--api-fallback-approved-gemini` (Gemini lane only)
  - `--api-fallback-approved-openai` (OpenAI lane only)
- Environment variables (programmatic / skill activation):
  - `JUDGE_PANEL_API_FALLBACK_APPROVED=1`
  - `JUDGE_PANEL_API_FALLBACK_APPROVED_GEMINI=1`
  - `JUDGE_PANEL_API_FALLBACK_APPROVED_OPENAI=1`

When approved, the relevant API key must also be set
(`GEMINI_API_KEY` / `GOOGLE_API_KEY` / `OPENAI_API_KEY`); otherwise the
juror returns `flags=[API_FALLBACK_NO_KEY]`. **API fallback bills the
user's pay-per-token API account, not the flat-fee Pro subscription** —
this is why approval is explicit and per-lane.

### Models used (pinned, read from `$CO_DIALECTIC_ENV` — default: `~/.co-dialectic/.env`)

| Stage | Model | Family | Role | Notes |
|---|---|---|---|---|
| Small-fish | `Gemini 3.5 Flash (Low)` | Google | Panel juror 1 | Antigravity OAuth / Ultra |
| Small-fish | `gpt-5.4` | OpenAI | Panel juror 2 | See OAuth-tier caveat below |
| Tiebreaker (default) | `Gemini 3.1 Pro (High)` | Google | Final verdict | Cross-tier vs. Flash |
| Tiebreaker (alt) | `gpt-5.4` | OpenAI | Final verdict | Pass via `--tiebreaker gpt-5.4` |

**OAuth-tier caveat (the small/big cascade collapses on the OpenAI lane).**
ChatGPT-account-auth Codex rejects `gpt-5.4-nano` and other API-only
nano/mini-tier models with: `"The 'gpt-5.4-nano' model is not supported
when using Codex with a ChatGPT account."` So the small-fish OpenAI juror
defaults to `gpt-5.4` (the cheapest ChatGPT-Plus-permitted tier). On the
OpenAI lane, small=big in tier — but the cross-FAMILY cascade still holds
(Gemini Flash vs. GPT-5.4 are different training distributions).
The default tiebreaker is therefore `Gemini 3.1 Pro (High)` — crossing
the tier boundary inside Google AND remaining cross-family vs. both small
judges and the Claude author.

**Override the small-OpenAI model** via `JUDGE_PANEL_OPENAI_OAUTH_MODEL`
(in `$CO_DIALECTIC_ENV` or shell env). When OpenAI ships a ChatGPT-permitted
mini/nano tier, set the pin and the small-OpenAI cost drops automatically.

**Cross-family guarantee:** the two small judges are from different
families (Google + OpenAI). The tiebreaker is cross-family vs. author
(Claude) AND cross-family vs. whichever small judge triggered escalation.
No two judges in the cascade share a training distribution with the author.

## Confidence + agreement rules

Each juror returns: `{"verdict": "pass|fail|uncertain", "confidence": 0-100, "flags": [...]}`.

**Agreement check:**
- Both small judges return `pass` → AGREE
- Both small judges return `fail` → AGREE
- One `pass`, one `fail` → DISAGREE (always escalates)
- One or both `uncertain` → DISAGREE (always escalates)

**Confidence check (only meaningful when AGREE):**
- Both confidences ≥ 80 → HIGH confidence (skip escalation)
- Either confidence < 80 → LOW confidence (escalate)

**Escalation condition:** `DISAGREE OR (AGREE AND LOW_CONF)`. The
small-fish panel verdict stands only on `AGREE AND HIGH_CONF`.

**Why ≥2 small-fish validation:** a single small judge's verdict is one
sample — it could share a blind spot with the author by coincidence.
Requiring two independent cross-family small judges to converge is a
Rastogi-2023-style complementation move: each juror's weakness is paired
with the other juror's orthogonal training distribution, so agreement is
structurally informative.

## Invocation — how the main agent runs it

The agent invokes the skill by calling the bundled TypeScript harness:

```
bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
  --rubric "<rubric slug or inline rubric>" \
  --artifact-file <path-to-artifact> \
  [--persona "<name(s)>"]
```

Or by passing the artifact inline:

```
bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
  --rubric hallucination \
  --artifact "The response text to evaluate..."
```

With an explicit persona override (overrides the rubric default):

```
bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
  --rubric spec-coherence \
  --artifact-file spec.md \
  --persona "Linus Torvalds"
```

Output is a single JSON object on stdout (schema below). No other stdout
writes — errors go to stderr. This makes the skill composable: any other
skill can call `judge_panel.ts`, parse stdout as JSON, and act on the
verdict without prompting the main LLM again.

## Output JSON shape

```json
{
  "version": "3.4.0",
  "rubric": "hallucination",
  "cascade": {
    "stage_1_small_fish": [
      {
        "model": "Gemini 3.5 Flash (Low)",
        "family": "google",
        "verdict": "pass",
        "confidence": 88,
        "flags": [],
        "tokens_in": 412,
        "tokens_out": 48,
        "latency_ms": 1840
      },
      {
        "model": "gpt-5.4",
        "family": "openai",
        "verdict": "pass",
        "confidence": 85,
        "flags": [],
        "tokens_in": 412,
        "tokens_out": 52,
        "latency_ms": 8200
      }
    ],
    "agreement": "agree",
    "confidence_tier": "high",
    "escalated": false,
    "stage_2_tiebreaker": null
  },
  "final_verdict": "pass",
  "final_confidence": 86,
  "cross_family": {
    "distinct_families_returned": 2,
    "degraded": false,
    "down_lanes": []
  },
  "all_flags": [],
  "cost_usd_estimate": 0.0041,
  "cost_vs_naive_parallel_jury_ratio": 0.32
}
```

When escalation fires, `stage_2_tiebreaker` is populated and
`final_verdict` comes from the tiebreaker, weighted by whichever small
judge agreed with it.

### Cross-family health contract

The output includes a top-level `cross_family` object:

```json
{
  "distinct_families_returned": 1,
  "degraded": true,
  "down_lanes": [
    {
      "family": "google",
      "model": "Gemini 3.5 Flash (Low)",
      "reason": "no JSON object in response"
    }
  ]
}
```

`distinct_families_returned` counts distinct families that returned a real
`pass`, `fail`, or `uncertain` verdict across every juror that ran
(small-fish plus tiebreaker when fired). `down_lanes` lists every juror lane
that errored, timed out, or returned empty/no-JSON output. `degraded` is true
when fewer than two distinct families returned a real verdict.

**Consumer contract:** gates MUST read `cross_family.degraded` as the
authoritative machine-checkable signal. NEVER infer degradation from
`all_flags[0]`; `all_flags` is for human skim and is not a positional API. A
degraded cross-family T3 result should be surfaced clearly, e.g.
`families: openai (google DOWN: no JSON object in response)`. The consumer
decides policy per stakes, including whether that degraded result is WARN or
BLOCK.

## Rubric slugs bundled with the skill

The TypeScript harness ships with named rubrics. A caller passes the slug;
the harness substitutes the artifact into the rubric template and sends
to each judge.

| Slug | Domain | What it evaluates |
|---|---|---|
| `hallucination` | Factual risk | Specificity, citation plausibility, confidence calibration, contradiction (mirrors codi-v3.1 spec §5.3) |
| `flattery` | Sycophancy | High/medium/low markers from calibration-auditor spec |
| `spec-coherence` | Artifact coherence | Blast-radius checks, claims-vs-implementation drift (P9) |
| `patent-safety` | Patent disclosure | §102 prior art risk, claim/spec boundary leakage |
| `prompt-quality` | Prompt rubric | Specificity, context, reasoning depth, intent clarity |
| `custom` | Any | Pass `--rubric-text <inline>` with the full rubric text |

Callers can pass `--rubric custom --rubric-text "..."` for ad-hoc rubrics.

## How to verify the skill works

**Trigger command 1 (explicit):** Say `judge-panel check this: "The
Eiffel Tower was built in 1923 and is located in Madrid."`

**Expected output:** Both small judges flag factual errors; AGREE + HIGH
confidence on `fail`; no escalation; JSON verdict with two `flags` entries
(wrong year, wrong city).

**Trigger command 2 (escalation):** Say `judge-panel check this:
"FrugalGPT (Chen et al. 2023) achieves 30-98% cost savings via a
cheap→expensive LLM cascade."`

**Expected output:** Both small judges likely `pass` with varying
confidence (one may flag the savings range as too wide or the claim as
unsupported without citation). If disagreement → escalates. JSON verdict
with `escalated: true`.

**Trigger command 3 (silent mode, as another skill):**
```
bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
  --rubric hallucination --artifact "..." --silent
```
**Expected output:** Pure JSON on stdout, nothing on stderr unless an
error fires.

**Failure modes:**
- Any judge call times out (> 30s) → mark that judge as `timeout`, proceed
  with remaining judges; if fewer than 2 small-fish returned, force escalate
- Rubric returns non-JSON from a judge → retry once; if second attempt
  also non-JSON, mark that judge as `parse_error` and force escalate
- All judges fail → return `final_verdict: "error"` with stderr log

## Cost discipline (P13 + Ground Zero 3D)

**Optimal cost, not minimum cost — and on OAuth, "cost" is bounded by
the user's flat subscription fee.** The `cost_usd_estimate` field still
reports what the run WOULD have cost on the pay-per-token API
(useful for cascade-vs-naive-jury comparison), but actual marginal cost
to the user is zero per call up to subscription quota.

OAuth tradeoffs the user is consciously accepting:
1. **Latency:** local-CLI calls add ~3-10s of process startup per call
   (codex spins up a session, agy starts an Antigravity run). Wall-clock
   per cascade is ~10-20s vs. ~2-5s for the API path. Cross-family
   verification is still in P11 / P21 budget.
2. **Subscription rate-limits:** ChatGPT Plus and Gemini Pro have
   per-day or per-hour usage caps. A judge-panel campaign that fires
   the cascade on hundreds of artifacts in a session can exhaust the
   cap. If a juror returns `verdict="error"` with `rate_limit` in the
   message, throttle the cascade or fall back to a different rubric.
3. **Tier collapse on OpenAI lane** (see Auth model section above).

If escalation fires on every run, the cost advantage collapses. If
the eval harness reports escalation_rate > 50% over 20 runs, the
confidence threshold (currently 80) is too strict — loosen it, or the
rubrics are ambiguous — sharpen them. This is P14 (self-evolution)
applied to the cascade.

## Auto-fire trigger (Protocol 8 — co-dialectic v4.1+)

**Protocol 8 (Auto-Verify by Stakes) dispatches to this skill automatically
at T3 and T4. No user command required.**

| Tier | Rubric selected | Escalation threshold | What user sees |
|---|---|---|---|
| T3 (important-decision/hard-to-undo) | `spec-coherence` for architectural/spec artifacts; `hallucination` for factual artifacts — caller picks based on artifact shape | Standard (agree+high-conf skips escalation) | `✓ reviewed by 2 models` on pass; `⚠ review flagged` on fail |
| T4 (irreversible/external-facing) | `hallucination` for general claims; biographical claims routed to `career-os.bio-claim-verifier` first (fallback to `hallucination` until that skill ships) | Standard | Result feeds into RED preflight summary; user must type 'send' to confirm |

**Integration contract:** Protocol 8 invokes judge-panel at T3 via the
fish-swarm dispatcher (same harness; fish-swarm rubric routes the T3 call
through the cascade). At T4, invoked directly for post-flight scoring.

**FAIL-HARD (T3 dispatch):** If zero fish are reachable when Protocol 8
triggers T3 auto-verify, the response is BLOCKED per the FAIL-HARD
invariant. Protocol 8 surfaces the fish-school remediation block. It does
NOT silently absorb the T3 cascade into Claude's own context — same-model
self-review defeats the cross-family guarantee this skill exists to enforce.

**Cost discipline for auto-fire:** Protocol 8 only reaches T3 on artifacts
where the classifier reaches HIGH or MEDIUM confidence that the artifact is
architectural/load-bearing/hard-to-undo. The vast majority of conversational
turns are T0/T1 and never reach judge-panel. Expected cascade rate: T3 fires
~5-15% of turns; T4 fires ~1-5%. Token budget is consistent with OPERATIONAL
DISCIPLINE right-sizing.

## Agent-teams quality gate (v4.22.0+)

When Claude Code agent teams are enabled (`CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1`),
the plugin's `TaskCompleted` hook (`hooks/task-completed-judge-gate.ts`) runs this
cascade on every teammate task completion — **opt-in via `CODI_TEAM_JUDGE_GATE=1`**.

- Rubric: `spec-coherence` by default (override: `CODI_TEAM_JUDGE_RUBRIC`)
- `fail`/`uncertain` verdict → task completion BLOCKED (hook exit 2); the
  jurors' flags are delivered to the teammate as actionable feedback
- FAIL-HARD when armed: if the cascade cannot run (CLIs missing/broken), the
  task is blocked with remediation — an unreviewable task never completes as
  if it were reviewed
- Tasks under 80 chars of content are skipped (nothing substantive to judge)

This makes the Independent Verification Gate structural for multi-agent work:
cross-family review fires on every task boundary, enforced by the harness, not
by agent discipline.

## Relationship to other skills

- **Upstream callers:** `hallucination-detector` (plugin #3),
  `calibration-auditor` (plugin #8 — for external-verdict mode),
  any skill that needs a cheap cross-family sanity check
- **Independent Verification Gate:** judge-panel is the default runtime
  mechanism for satisfying the Gate on artifacts where full human +
  model-diversity review is too slow or expensive
- **Anti-pattern:** do NOT call judge-panel from inside a Claude subagent
  spawned by another Claude. Same-model subagent would pollute the
  cross-family guarantee. Always invoke from the main agent or directly
  from the Python harness

### END JUDGE-PANEL ###
