---
title: "fix: Add Proof-of-Behavior Verification Phase to Printing Press"
type: fix
status: active
date: 2026-03-26
---

# fix: Add Proof-of-Behavior Verification Phase to Printing Press

## Overview

The printing press generates CLIs that score Grade A on its own scorecard but fail on the first real API call. The root cause is Goodhart's Law: the scorecard measures **string presence** ("does sync.go contain `readCache`?") not **behavior** ("does sync.go actually call `UpsertMessage` on a table that search.go queries?"). Claude, optimizing for the scorecard, writes dead code containing trigger strings.

Evidence from 3 shipped CLIs:
- **Discord CLI**: Scored 96/110, but sync.go hit flat paths (`GET /messages`) that don't exist in Discord's guild-scoped API. Store created domain tables nothing populated. Auth sent `Bearer` instead of `Bot`.
- **Notion CLI**: Scored 79/100, but `members_fts` FTS5 table was populated during sync but never queried by any command. Empty `if flags.timeout > 0 {}` block exists purely to satisfy dead-code checker. `rateLimitErr()` called in forecast.go on a non-rate-limit error path.
- **Linear CLI**: Scored 96/110, but `--csv` and `--stdin` flags registered and never checked. Helper functions existed purely to trigger scorecard matches.

## Problem Statement

The current pipeline has verification scattered across three places that don't talk to each other:

| Location | What it checks | Why it fails |
|----------|---------------|-------------|
| `validate.go` (7 gates) | `go build`, `go vet`, binary runs, `--help`/`version`/`doctor` produce output | Only checks compilation - doesn't verify behavior |
| `dogfood.go` (5 checks) | Path regex, auth format, dead flags/functions, pipeline integrity | String-matching: checks if `UpsertX` appears in source, not if it's on a live code path |
| `scorecard.go` Tier 2 (6 dims) | PathValidity, AuthProtocol, DataPipeline, SyncCorrectness, TypeFidelity, DeadCode | Same string-matching problem. Plus scoring 0-10 makes failures gradual when they should be binary |

**The missing piece**: No check verifies that the **data actually flows**. You can have:
- A `UpsertMessage()` method in store.go (scorecard: +2 points)
- A `sync.go` file that imports store (scorecard: +2 points)
- But sync.go calls generic `Upsert()`, not `UpsertMessage()` (scorecard doesn't catch this)
- So the domain tables are ghost tables - created but never populated

## Proposed Solution

Replace the scattered checks with a single **Proof-of-Behavior** phase that runs as a hard gate after Phase 4 (GOAT Build) and before Phase 5 (Final Scorecard). This phase produces a structured `VerificationReport` that the scorecard consumes.

### Architecture

```
Phase 4 (GOAT Build)
        |
        v
Phase 4.7: PROOF OF BEHAVIOR  <-- NEW
  |-- 4.7a: Compile Gate (go build + go vet)
  |-- 4.7b: Path Proof (dry-run URL validation against spec)
  |-- 4.7c: Flag Proof (every registered flag referenced in commands)
  |-- 4.7d: Pipeline Proof (every domain table has WRITE + READ + SEARCH paths)
  |-- 4.7e: Auth Proof (auth header format matches spec's securitySchemes)
  |-- 4.7f: Verdict (PASS/WARN/FAIL with structured report)
  |-- 4.7g: Auto-Remediation (safe deletions only, then re-verify)
        |
        v
Phase 5 (Final Scorecard - consumes VerificationReport)
```

### Key Design Decisions

1. **Hard gate, not a score.** The current scorecard gives 0-5 points for dead code. This means 3 dead flags still passes. The Proof-of-Behavior phase is pass/fail: zero dead flags, zero ghost tables, zero hallucinated paths, or it doesn't ship.

2. **Grep-based analysis, not `golang.org/x/tools/cmd/deadcode`.** The external `deadcode` tool uses Rapid Type Analysis from `main()`. In a Cobra app, all RunE functions appear unreachable because Cobra dispatches via runtime reflection. The existing grep-based approach in `scorecard.go:1031-1064` is Cobra-aware and purpose-built for this codebase.

3. **Auto-fix is deletion-only.** Safe auto-fixes: delete dead functions, delete unwired flags, delete ghost tables. Unsafe operations (wiring `sync.go` to call `UpsertMessage` instead of generic `Upsert`) require re-running Phase 4's GOAT Build, not a patch. The line between "fix" and "re-generate" is: if you need to write new code, it's re-generation.

4. **Compile gate after every auto-fix.** After removing dead code, `go build ./... && go vet ./...` must pass before re-verification. If auto-fix breaks compilation, auto-fix is reverted and the failure is reported as FAIL (not WARN).

5. **User-edited code protection.** Files without the `// Code generated by CLI Printing Press` header are never auto-fixed. Files with uncommitted modifications (checked via `git diff`) are flagged but not touched.

## Technical Approach

### Phase 4.7a: Compile Gate

```go
// internal/pipeline/verify.go
func (v *Verifier) CompileGate() error {
    // Same 3 gates as validate.go but re-run after GOAT Build
    // 1. go mod tidy
    // 2. go vet ./...
    // 3. go build ./...
    // If any fail, Phase 4.7 is FAIL - no point checking behavior
}
```

### Phase 4.7b: Path Proof

For each generated command file in `internal/cli/`:
1. Extract the URL path from the source code (regex: `path\s*[:=]\s*"([^"]+)"` or `c\.(Get|Post|Patch|Delete|Put)\s*\(\s*"([^"]+)"`)
2. Normalize path params: replace `/guilds/123/channels` with `/guilds/{guild_id}/channels`
3. Look up the normalized path in the original OpenAPI spec's `paths` object
4. If the path doesn't exist in the spec: **CRITICAL** - hallucinated endpoint

```go
// internal/pipeline/verify.go
type PathProofResult struct {
    Command      string // e.g., "guilds channels list"
    ExtractedURL string // e.g., "/guilds/{guild_id}/channels"
    Method       string // e.g., "GET"
    InSpec       bool
    SpecPath     string // matched spec path, empty if not found
}

func (v *Verifier) PathProof(specPaths map[string]map[string]bool) []PathProofResult {
    // 1. Glob internal/cli/*.go (exclude root.go, helpers.go, doctor.go, auth.go, version.go)
    // 2. For each file, extract path assignments via regex
    // 3. Skip files with no path (local-only commands: search, sql, activity, stale, etc.)
    // 4. Normalize extracted path (replace literal IDs with {param} placeholders)
    // 5. Match against specPaths map
    // 6. Return results with InSpec=false for hallucinated paths
}
```

**GraphQL exception**: For GraphQL APIs (detected by single `/graphql` endpoint in spec), skip path validation entirely. Instead, validate that query strings in generated code reference real schema types (parse `.graphql` query files against schema).

**Local-command exception**: Commands that only query local SQLite (search, sql, activity, stale, trends, patterns, forecast, health, similar, bottleneck, status) don't have paths. Skip them. Detection: file has no `c.Get/Post/Patch/Delete/Put` calls.

### Phase 4.7c: Flag Proof

For every persistent flag registered in `root.go`:
1. Parse the flag field name from `rootCmd.PersistentFlags().XxxVar(&flags.fieldName, ...)`
2. Search all files in `internal/cli/*.go` EXCLUDING `root.go` for `flags.fieldName`
3. Also search `internal/client/client.go` for `f.fieldName` or `flags.fieldName` (catches flags passed through newClient)
4. If a flag has zero references outside its declaration: **dead flag**

```go
type FlagProofResult struct {
    FlagName   string // e.g., "dryRun"
    CLIName    string // e.g., "--dry-run"
    RefCount   int    // references in command files + client
    DeadFlag   bool
}
```

**Indirect usage rule**: A flag counts as "used" if:
- It appears in a command file's RunE function (`flags.dryRun`)
- OR it's assigned to the client config struct in `newClient()` AND the client field is checked in `client.go`
- OR it's checked in `helpers.go` AND the helper is called from a command file

This catches the `flags.timeout -> c.Timeout -> client uses it` pattern.

### Phase 4.7d: Pipeline Proof (the critical one)

For every domain-specific SQLite table in `store.go`:

```go
type PipelineProofResult struct {
    TableName   string // e.g., "messages"
    HasWrite    bool   // sync.go calls UpsertMessage or store.UpsertMessage
    WritePath   string // e.g., "sync.go:142"
    HasRead     bool   // any command file queries this table
    ReadPath    string // e.g., "activity.go:58"
    HasSearch   bool   // if FTS5 table exists, a SearchX() is called
    SearchPath  string // e.g., "search.go:54"
    FTSExists   bool   // FTS5 virtual table exists for this entity
    GhostTable  bool   // table created but never populated
    OrphanFTS   bool   // FTS5 exists but SearchX never called
}
```

Detection algorithm:
1. Parse `store.go` for `CREATE TABLE` statements -> extract table names
2. Parse `store.go` for `CREATE VIRTUAL TABLE ... USING fts5` -> extract FTS table names and their source tables
3. For each domain table (exclude `sync_state`, `resources`, `*_fts`):
   a. WRITE: grep `internal/cli/*.go` for `UpsertTableName` or `store.UpsertTableName` or `db.UpsertTableName`
   b. READ: grep `internal/cli/*.go` for `FROM tablename` or `JOIN tablename` or table-specific query methods
   c. SEARCH: if FTS table exists, grep for `SearchTableName` or `tablename_fts MATCH`
4. A table with HasWrite=false is a **ghost table** (created but never populated)
5. A table with HasWrite=true but HasRead=false is an **orphan table** (populated but never queried)
6. An FTS table with no SearchX calls is an **orphan index**

**Exempt tables**: `sync_state` (internal metadata - read/written by sync only), `resources` (generic fallback - acceptable if domain tables exist), `*_fts` (populated by triggers, read by search).

### Phase 4.7e: Auth Proof

Compare the generated auth header against the spec's `securitySchemes`:

```go
type AuthProofResult struct {
    SpecFormat     string // e.g., "Bot {token}" or "Bearer {token}"
    GeneratedFormat string // extracted from client.go
    Match          bool
    EnvVarCorrect  bool   // env var name matches spec convention
}
```

Detection:
1. Parse spec's `securitySchemes` for auth type (apiKey, http bearer, http basic, oauth2)
2. Parse `internal/client/client.go` for the Authorization header value format
3. Compare: if spec says `Bot {token}` but code sends `Bearer {token}`, that's a **critical mismatch** (every request will 401)

### Phase 4.7f: Verdict

```go
type VerificationReport struct {
    Paths      []PathProofResult
    Flags      []FlagProofResult
    Pipeline   []PipelineProofResult
    Auth       AuthProofResult

    // Computed
    HallucinatedPaths int  // paths not in spec
    DeadFlags         int
    GhostTables       int
    OrphanFTS         int
    AuthMismatch      bool

    Verdict    string // "PASS", "WARN", "FAIL"
    // FAIL if: any hallucinated paths, auth mismatch, or ghost tables on primary entities
    // WARN if: dead flags > 0, orphan FTS, ghost tables on support entities only
    // PASS if: everything clean
}
```

**FAIL** = hard stop. Cannot proceed to Phase 5. Must fix before shipping.
**WARN** = proceed but flag in final report. These are quality issues, not correctness issues.
**PASS** = clean. Full confidence in generated CLI.

### Phase 4.7g: Auto-Remediation

Only runs if verdict is WARN or FAIL. Only does **safe deletions**:

| Issue | Auto-Fix | Safety |
|-------|----------|--------|
| Dead flag in root.go | Remove the `PersistentFlags().XxxVar` line + struct field | Safe - compilation proves correctness |
| Dead function in helpers.go | Remove the function | Safe - if anything called it, `go build` will fail |
| Ghost table (never populated) | Remove CREATE TABLE from store.go | Safe - nothing references it |
| Orphan FTS (never searched) | Remove CREATE VIRTUAL TABLE + triggers | Safe - nothing queries it |
| Hallucinated path | **DO NOT AUTO-FIX** - flag for GOAT Build re-run | Unsafe - needs domain knowledge |
| Auth mismatch | **DO NOT AUTO-FIX** - flag for manual review | Unsafe - wrong fix = all requests fail |
| Broken data pipeline (sync calls generic Upsert) | **DO NOT AUTO-FIX** - flag for GOAT Build re-run | Unsafe - needs code generation |

After auto-fix:
1. Run `go build ./... && go vet ./...` - if fails, revert all auto-fixes and report FAIL
2. Re-run all proofs (4.7b-4.7e)
3. If still FAIL after auto-fix: report as FAIL with "auto-fix insufficient, manual intervention required"
4. Maximum 1 auto-fix cycle (no recursive fixing)

### Anti-Gaming Rules

These rules go in the SKILL.md to prevent Claude from hallucinating to optimize for the new checks:

```
PROOF-OF-BEHAVIOR RULES:

1. NEVER add a function to helpers.go that isn't called by a real command.
   The Proof-of-Behavior phase traces every function to a call site.

2. NEVER register a flag in root.go that no command checks.
   The flag proof traces every flag to a usage site in RunE code.

3. NEVER create a SQLite table that sync doesn't populate.
   The pipeline proof traces every table to a UpsertX call in sync.go.

4. NEVER create an FTS5 index that no search command queries.
   The pipeline proof traces every FTS5 table to a SearchX call.

5. NEVER add an empty if-block to satisfy dead-code detection.
   `if flags.timeout > 0 { // already applied }` is still dead code.
   The flag must be READ (its value influences control flow or output).

6. NEVER call a function in the wrong context to satisfy wiring checks.
   `rateLimitErr()` called on a non-rate-limit path is a hallucination.
   The function must be called where its name implies it should be called.
```

## Acceptance Criteria

### Must Have (FAIL gate)
- [ ] Zero hallucinated API paths (every extracted URL exists in the spec)
- [ ] Zero auth mismatches (generated auth format matches spec's securitySchemes)
- [ ] Zero ghost tables on primary entities (every primary domain table has a WRITE path)
- [ ] Compile gate passes after auto-remediation
- [ ] Structured `VerificationReport` produced as JSON + markdown

### Should Have (WARN gate)
- [ ] Zero dead flags (every root.go flag referenced in command or client code)
- [ ] Zero dead functions (every helpers.go function called from a command file)
- [ ] Zero orphan FTS indexes (every FTS5 table queried by a search command)
- [ ] Auto-remediation deletes dead flags/functions/tables and re-verifies

### Nice to Have
- [ ] GraphQL query field validation (for non-REST APIs)
- [ ] User-edit protection (skip auto-fix on files without generated-code header)
- [ ] Scorecard Tier 2 consumes VerificationReport instead of running its own checks

## System-Wide Impact

- **Interaction graph**: Phase 4.7 reads the OpenAPI spec (loaded in Phase 0), reads generated source files (from Phase 2-4), and produces a VerificationReport consumed by Phase 5's scorecard. Auto-remediation modifies generated files, requiring re-compilation.
- **Error propagation**: If auto-fix breaks compilation, all fixes are reverted atomically (git stash or copy-restore). No partial fix state.
- **State lifecycle risks**: Auto-fix could delete a function that a user manually added after generation. Mitigated by the generated-code header check.
- **API surface parity**: The `printing-press scorecard` command should gain a `--verify` flag that runs the Proof-of-Behavior phase standalone.

## Implementation Phases

### Phase 1: Verification Engine (core)
- [ ] `internal/pipeline/verify.go` - `Verifier` struct with `PathProof()`, `FlagProof()`, `PipelineProof()`, `AuthProof()`
- [ ] `internal/pipeline/verify_report.go` - `VerificationReport` struct with JSON/markdown serialization
- [ ] Wire into `fullrun.go` between GOAT Build and Final Scorecard
- [ ] Tests: `verify_test.go` with fixtures from Discord/Notion/Linear CLIs

### Phase 2: Auto-Remediation
- [ ] `internal/pipeline/remediate.go` - safe deletion functions (remove dead flag, remove dead function, remove ghost table)
- [ ] Compile gate after remediation
- [ ] Re-verification loop (max 1 cycle)
- [ ] Tests: `remediate_test.go` with before/after fixtures

### Phase 3: Scorecard Integration
- [ ] Replace `scoreDeadCode()`, `scorePathValidity()`, `scoreDataPipelineIntegrity()` with VerificationReport consumption
- [ ] Add `--verify` flag to `printing-press scorecard` command
- [ ] Update SKILL.md Phase 4.5/4.6 to reference Phase 4.7

### Phase 4: Anti-Gaming Rules
- [ ] Add Proof-of-Behavior rules to SKILL.md
- [ ] Add rule 5 detection: empty if-blocks that reference a flag but don't use its value
- [ ] Add rule 6 detection: function called in semantically wrong context (heuristic: function name implies error type but caller doesn't handle that error type)

## Success Metrics

| Metric | Before | Target |
|--------|--------|--------|
| Hallucinated paths caught before ship | 0% (found manually after) | 100% |
| Dead flags in shipped CLIs | 3-5 per CLI | 0 |
| Ghost tables in shipped CLIs | 1-2 per CLI | 0 |
| Auth mismatches caught before ship | 0% (found on first API call) | 100% |
| Time to verify | N/A (manual) | < 30 seconds per CLI |

## Dependencies & Risks

- **Risk**: Auto-fix could be too aggressive and delete functions that are "dead" only because the call site hasn't been generated yet (in a multi-phase build). Mitigation: Auto-fix only runs AFTER the GOAT Build phase completes.
- **Risk**: Path normalization (replacing literal IDs with `{param}`) could fail on unusual URL patterns. Mitigation: Use the spec's path templates as the source of truth, match generated paths against them with regex.
- **Risk**: False positives on "indirect" flag usage through struct assignment chains. Mitigation: Follow the assignment chain through `newClient()` and the client struct.

## Sources & References

### Internal References
- Existing dogfood validator: `internal/pipeline/dogfood.go`
- Existing scorecard Tier 2: `internal/pipeline/scorecard.go:800-1080`
- Existing self-improvement loop: `internal/pipeline/selfimprove.go`
- Existing quality gates: `internal/generator/validate.go`
- Discord CLI hallucination evidence: `docs/plans/2026-03-26-feat-printing-press-v2-anti-hallucination-overhaul-plan.md`

### External References
- Go deadcode tool (why we DON'T use it for Cobra apps): https://go.dev/blog/deadcode
- kin-openapi spec validation: https://github.com/getkin/kin-openapi
- oapi-codegen compile-time interface proof pattern: https://github.com/oapi-codegen/oapi-codegen
- Speakeasy's generate-compile-test loop: https://www.speakeasy.com/docs/sdks/sdk-contract-testing/bootstrapping-test-generation
