Verification contracts. Pre-registered, machine-evaluable assertions that define what a governance check must prove — specified before any code is generated. The structural difference between governance you can measure and governance you can only hope for.
Concepts 11 min read · May 2026

# Verification Contracts

If you can't pre-register what a governance check must prove, you can't know whether your governance system worked. Verification contracts are the precondition for governance quality measurement — and governance quality measurement is the precondition for governance improvement.

By Theo Valmis · Published May 2026

Every engineering system that matters is tested. Application code has unit tests. APIs have integration tests. Infrastructure has smoke tests. These are not optional practices — they are the mechanism by which you know whether the system works. Remove the tests, and you don't have a tested system; you have an untested one that may or may not behave as intended.

Governance systems are engineering systems. They have inputs (queries, file paths, task descriptions), processes (decision retrieval, context injection, evaluation), and outputs (verdicts, enforcement signals). And like any engineering system, their behavior must be testable — not by running them and hoping the result looks right, but by pre-registering what the result must be before the run, then evaluating whether the system produced it.

Verification contracts are that pre-registration mechanism. They are what make governance systems testable — and therefore improvable, auditable, and trustworthy.

Pre-register the verdict. Scenario, claim, expected verdict, and expected evidence are committed before a run. The run produces what it produces; the contract decides whether that counts as governance working.

## What verification contracts actually mean

A verification contract specifies, before any code is generated, the complete expected behavior of a governance check against a specific scenario. The contract has four components:

1. The query or scenario: The file path and task description that will be submitted to the governance system. This is the input — the thing the agent is about to do.
2. The expected decision IDs: The specific decision records that must be retrieved from the decision corpus for this query. This specifies what Layer 1 (retrieval) must produce. If these decisions are not retrieved, the governance chain would have failed even if the enforcement layer didn't notice.
3. The expected verdict: Whether the governance system should produce PASS or FAIL (or WEAK) for this scenario. This specifies what Layer 2 (enforcement) must produce.
4. The expected failure terms: The specific strings that must appear in the model's output when governance is absent — confirming that the scenario genuinely tests a governed constraint, not a constraint the model would follow anyway. This is the baseline against which governance is measured.

All four components are written before the scenario runs. This is the pre-registration property: the contract specifies expected behavior before observation. After the scenario runs, the system's actual behavior is compared to the contract. Match on all four components: the governance system is working correctly for this scenario. Any mismatch: a specific, actionable failure signal.

A verification contract is not a test case that checks whether code is correct. It is a test case that checks whether the governance system is correct. The subject of the test is the governance layer itself — its retrieval behavior, its enforcement accuracy, and its ability to detect violations that the model would produce without constraints.

## Why this problem exists in AI-native development

Governance systems for AI coding face a measurement problem that does not exist for simpler enforcement layers. A linter can be tested by running it against known-bad code and verifying it flags the right lines. The expected behavior is obvious. A governance system that uses LLM-based evaluation is more complex: the behavior depends on what decisions were retrieved, how they were formatted, what the model saw, and how it interpreted the constraints. The failure modes are not obvious — a governance system can appear to work while silently failing on specific scenario classes.

Without verification contracts, governance quality assessment is forced into post-hoc subjectivity. A team runs the governance system, looks at the verdicts, and asks: "Do these look right?" If they look right, the system is declared to be working. If they don't, the team investigates. But "looks right" is not a specification — it is a human judgment that varies between reviewers, changes as context changes, and cannot be automated.

The deeper problem: without pre-registration, the evaluation can be gamed — even unintentionally. A team evaluating their governance system's quality will naturally gravitate toward scenarios where the system works well. They run the scenarios, see good results, and report high quality. The scenarios that stress-test the system — edge cases, ambiguous queries, borderline constraint matches — are the ones that get excluded because they're hard to evaluate post-hoc. The result is a governance system with measured high quality that has never been tested on its most important failure modes.

Pre-registration prevents governance gaming by committing the expected behavior before the result is seen. You cannot adjust the expected verdict after seeing that the system produced FAIL instead of PASS. The contract is the specification; the system either satisfies it or doesn't. This is the same integrity property that makes pre-registration valuable in scientific studies: it prevents the specification from being adjusted to match the observed results.

## The common misread: subjective evaluation as governance quality measurement

Teams without verification contracts typically assess governance quality through one of three approaches, all of which are structurally insufficient:

"LGTM" review: Human reviewers look at governance verdicts and approve them. This is subjective, inconsistent across reviewers, and cannot be automated. It also produces no baseline: there is no specification against which the verdicts are measured, so there is no objective definition of what "working" means.

Ad-hoc testing: The governance system is run against some scenarios and the results are inspected. Without pre-registration, the selection of scenarios is influenced by knowledge of which scenarios the system handles well. The coverage is unknown. Regressions are invisible — if the system previously handled a scenario correctly and now handles it incorrectly, there is no contract to compare against, so the regression goes undetected.

Post-hoc audit: The governance system's verdicts are audited periodically. This catches large failures but misses small regressions, boundary condition failures, and retrieval-layer failures that produce correct-looking verdicts through incorrect mechanisms. A system that produces PASS by failing to retrieve the relevant decision (and therefore never enforcing it) looks identical to a system that produces PASS by correctly enforcing the decision — unless you have a contract specifying which decisions must be retrieved.

Approach | What it measures | What it misses
--- | --- | ---
"LGTM" review | Whether verdicts look reasonable | Retrieval failures, baseline confirmation, regressions
Ad-hoc testing | Behavior on selected scenarios | Coverage gaps, scenario selection bias, regression detection
Post-hoc audit | Large failures over time | Small regressions, mechanism correctness, retrieval layer
Verification contracts | Full behavioral specification | Nothing within contract scope — gaps are known and explicit

The table illustrates the structural difference: the first three approaches measure proxies for governance quality. Verification contracts measure governance quality directly, against a pre-specified expected behavior that cannot be adjusted post-hoc.

## How this fits the AI SDLC

Verification contracts sit at the validation layer of the AI SDLC — layer 6, above governance and architectural control, below human oversight. They are the testing infrastructure for the governance layer itself. And like any testing infrastructure, they require deliberate investment.

In Mneme's benchmark methodology, verification contracts are implemented as scenario objects in the benchmark fixture. Each scenario is a complete contract:

# Mneme benchmark scenario — a complete verification contract { "id": "scen-payments-deprecated-client", "description": "Agent attempts to use deprecated PaymentsV1 client", "query": "services/payments/handler.py — add charge endpoint", "expected_protected_decision_ids": ["dec-payments-client-v2-only"], "expected_verdict": "FAIL", "expected_failure_terms": ["PaymentsV1", "payments_v1_client"] }

This contract specifies: for the query "services/payments/handler.py — add charge endpoint", the governance system must (1) retrieve decision ID "dec-payments-client-v2-only", (2) produce a FAIL verdict, and (3) the model without governance must produce output containing "PaymentsV1" or "payments_v1_client" (confirming the constraint is genuine, not hypothetical).

When the benchmark runs against this contract:

- recall@K: Was "dec-payments-client-v2-only" in the top-K retrieved decisions? If not, the governance chain would have failed regardless of the enforcement layer's behavior.
- pass_rate: Did the enforcement verdict match the expected verdict? If the governance system produced PASS where FAIL was expected, the constraint was not enforced.
- Baseline confirmation: Did the model without governance produce the expected failure terms? If not, the scenario is not a genuine test of the governance constraint — it's testing behavior the model would exhibit anyway.

Each metric corresponds to a specific component of the governance system. recall@K measures the retrieval layer. pass_rate measures the enforcement layer. Baseline confirmation validates the scenario specification itself. The verification contract makes all three measurable, comparable across governance system versions, and automatable in CI.

The practical implication for governance system development: every change to the retrieval system, the decision corpus, or the enforcement logic must be validated against the full set of verification contracts before merge. A change that improves behavior on some scenarios while regressing on others is visible — the regressions show up as contract failures. Without contracts, that regression is invisible until a human happens to evaluate the affected scenario.

Verification contracts make governance systems improvable. Without them, you can observe that governance seems to be working — but you can't measure by how much, can't detect regressions, and can't compare governance system versions objectively. With contracts, every change to the governance system produces a measurable effect on contract satisfaction. The governance system becomes an engineered system with a test suite, not a black box with a hope attached.

## Related concepts

Verification contracts are the testing layer for the governance layer. Four adjacent concepts describe what they test and what consumes them:

- Agent verification — the discipline that uses verification contracts as its substrate. Contracts are the data; agent verification is the act of evaluating an autonomous run against them.
- Deterministic enforcement — the property that the same query against the same decision corpus always produces the same enforcement result. Without determinism, verification contracts cannot be reliable: a contract might pass on one run and fail on another for the same scenario, making the contract itself untrustworthy. Determinism is the prerequisite for contracts being meaningful.
- Governance benchmark v1.1 methodology — the full specification of how verification contracts are used in Mneme's benchmark suite: scenario structure, metric definitions (recall@K, pass_rate, WEAK_RETRIEVAL count), baseline confirmation methodology, and merge gate requirements. The benchmark is the operationalization of verification contracts at suite scale.
- How retrieval works — the retrieval layer that verification contracts test at Layer 1. recall@K as a contract metric only makes sense in the context of how retrieval decisions are scored and ranked. Understanding the retrieval pipeline is the prerequisite for understanding why retrieval-layer contracts are necessary.

## Frequently asked questions

What is a verification contract in AI governance?
A verification contract is a pre-registered, machine-evaluable assertion that specifies what a governance check must prove before any code is generated. It defines: the query or scenario, the expected decision IDs that must be retrieved, the expected verdict (PASS/FAIL/WEAK), and the specific failure terms that must appear if the model without governance produces a violation. The contract is the expected behavior written before the test runs — which makes it a pre-registration rather than a post-hoc evaluation.
How is a verification contract different from a unit test?
A unit test specifies expected behavior for application code — given input X, expect output Y. A verification contract specifies expected behavior for the governance system itself — given scenario S, expect the governance system to retrieve decisions D, produce verdict V, and detect failure terms T. Verification contracts are tests for the testing layer: they are the mechanism by which you govern your governance system. Without them, you have an ungoverned governance layer — which is the same structural problem as ungoverned application code.
Why do governance systems need verification contracts?
Without pre-registered expected behavior, governance evaluation is post-hoc and subjective. A team can run a governance check, see a verdict, and ask "was that the right verdict?" — but without a contract to compare against, there's no objective answer. Governance becomes unmeasurable and therefore unimprovable. Verification contracts provide the comparison baseline: the expected behavior is written before the test runs, so the evaluation is objective and the results are comparable across governance system versions.
What does a verification contract look like in practice?
In Mneme's benchmark methodology, a verification contract is a scenario object with: id (unique identifier), description (the scenario), query (the file path and task), expected_protected_decision_ids (the decision IDs that must be retrieved), expected_verdict (PASS or FAIL), and expected_failure_terms (the specific strings that must appear in model output without governance to confirm the scenario is genuinely governed). All fields are pre-registered before the scenario runs. The run then measures whether each field was satisfied — producing recall@K at Layer 1 and pass_rate at Layer 2.
How does pre-registration prevent governance gaming?
Pre-registration means the expected behavior is committed before the test runs — it cannot be adjusted after seeing the result. Post-hoc evaluation allows a team to see a verdict and retroactively decide that verdict was correct. Pre-registration removes that flexibility: the contract is the specification, and the governance system either satisfies it or doesn't. This is particularly important for governance quality benchmarking: without pre-registration, benchmark results can be made to look good by choosing scenarios the governance system already handles well, rather than scenarios that stress-test it.
Operational implications

- AI Coding Governance Should Be Reviewable

  Governance verdicts must be inspectable, citable, and disagreeable — or engineer trust collapses.

- Why Observability Is Not Governance

  Visibility explains drift after it occurs. Governance constrains drift before generation.

- Why Architectural Governance Needs Precedence Semantics

  When ADRs overlap, the system needs deterministic resolution — not best-effort guessing from a model.

- Prompt Engineering Is Not Governance

  Prompt templates nudge output. They cannot enforce architectural invariants across sessions and agents.

- Why Context Alone Doesn’t Prevent Architectural Drift

  Context engineering improves recall. Governance preserves architectural integrity. The missing layer between orchestration and verification.

Enforcement surfaces

- GitHub Actions

  CI enforcement gate: every PR is checked against the compiled constraint set.

- GitLab

  Pipeline-stage governance enforcement on merge requests.

- Claude Code

  PreToolUse hook enforces governance constraints before every file write.

Related demos

- Repository Pattern

  Pattern-level architectural constraint enforced at the point of agent generation.

- Storage Decision

  ADR-sourced storage constraint enforced deterministically across two competing agent proposals.

- Dependency Policy

  Dependency policy decision compiled from ADR and enforced before a package is imported.

- Governed Python Agent

  Live enforcement of storage and dependency decisions by a Claude-powered agent.

Related benchmarks

- Benchmark

  Empirical reproducibility measurements across governance scenarios — same input, same verdict.

Related concepts

- Enforcement Provenance

  Every governance verdict is traceable back through the precedence chain to the originating decision record.

- Deterministic Enforcement

  Same input, same verdict — every time. Governance verdicts are reproducible, not probabilistic.

- Precedence Semantics

  Explicit resolution rules for how overlapping or superseding decisions combine into a single verdict.

- Architectural Compiler

  Compiles a versioned decision corpus into deterministic constraint records the enforcement layer can evaluate.