Evidence Pack

0. How Evidence Is Produced — and Why There Are Four Figures

Everything below this section is a number. This is where they come from, and why collapsing them into one “accuracy” figure would misrepresent all four.

The four populations, with live numbers

The same four bodies of evidence the model above describes, measured at generation time rather than transcribed. They are kept apart deliberately: a detection rate on hand-authored scenarios is not a conformance pass rate, and neither is a measure of production performance.

1. System Behavior Guarantees

2. Real-Corpus Detection Rates

The CTS YAML suite validates the policy engine against synthetic, hand‑authored cases. To check behavior against real conversational phrasing, the engine is also replayed against the Fabricate Battle‑Test Corpus — 550 conversations, of which 127 are labelled scenario_type=compliant.

Detection and false positives are measured on disjoint populations, per conversation — not per turn. Detection is the share of conversations declaring a given violation in which that control fired. The false‑positive rate is the share of the 127 compliant conversations in which it fired anyway.

ATR-01 is not listed because this corpus cannot measure it. The replay harness drops all 14 ATR‑01 expectations as untestable — audit-field completeness is a property of the live event envelope, not of a replayed transcript. That is a limitation of the measurement, not a score of zero. ATR-01 remains one of the five canonical controls; the supplemental rule is bot‑to‑bot.

The harness also drops 29 PDX‑01 expectations where disclosure occurs at turn 0, because the probe is then post‑disclosure and there is nothing to gate. The 41/41 above is measured after those are removed.

Self-reported and not independently audited. These are replay measurements against a fixed corpus, not a conformance claim and not evidence of field accuracy. DBC-01 misses 17 of 200 declared cases and fires on 5 of 127 compliant ones; deception heuristics remain the least precise of the controls and an active area of work.

3. Anonymized Failure Trace Example

The example below is synthetic — constructed from observed behavior patterns, with all identifying information removed. It shows what an IDG-01 (late disclosure) violation looks like in the NHID-Clinical audit trace, and what a payer auditor would see when reviewing it.

Anonymized Failure Trace — IDG-01 Violation
Source: Synthetic example based on observed behavior patterns. No real PHI, no real provider data.
Generated: 2026-06-07 | Policy version: nhid-clinical-v1.3 | Correlation ID: [REDACTED]

t=00:00.000  INGEST      POST /voice/process received
             session_id: [REDACTED]
             call_sid:   [REDACTED]
             caller_type: ai_agent

t=00:00.084  VALIDATE    SpeechResult normalized
             turn_count: 0
             content_hash: [REDACTED]

t=00:00.091  STATE       Session reconstructed
             turn_count: 0
             disclosure_timestamp: null
             disclosure_confirmed: false

t=00:00.098  POLICY      IDG-01 evaluated
             rule: "Disclose AI identity before any data exchange"
             turn_count: 0
             disclosure_confirmed: false
             trigger: FIRST_TURN_NO_DISCLOSURE
             action: DISCLOSE_IDENTITY
── Violation recorded ──────────────────────────────────────────────
t=00:00.103  VIOLATION   IDG-01
             severity: critical
             message: "AI identity not disclosed at call start"
             action_taken: DISCLOSE_IDENTITY (forced)
             data_exchanged_before_disclosure: false
             recoverable: true
────────────────────────────────────────────────────────────────────
t=00:00.109  EXEC        TwiML rendered — forced disclosure statement
             text: "This call is being handled by an automated system on behalf of [Provider Name Redacted]."
             disclosure_forced: true

t=00:00.114  PERSIST     Event written
             disclosure_timestamp: 00:00.109
             boundary_violations: ["IDG-01"]
             partial_failure: true
             deterministic_hash: [REDACTED]

── What this means ─────────────────────────────────────────────────
The AI agent did not disclose its automated nature at call start.
The policy engine detected a turn_count=0 exchange with no prior
disclosure and forced a disclosure statement before any data could
be shared. The violation is logged as critical but recoverable.
A payer auditing this session would see:
  - disclosure_timestamp set 109ms into the call (forced, not voluntary)
  - partial_failure: true
  - boundary_violations: ["IDG-01"]
────────────────────────────────────────────────────────────────────

4. Failure & Attack Simulation Coverage

The failure injection harness covers the following scenarios:

5. Audit Readiness Model

An external auditor reconstructing a session from the event store can determine:

• When the call started and when the first disclosure statement was made
• Whether disclosure preceded any PHI or credential exchange
• Whether opt-out or escalation was requested and how it was handled
• Which policy engine version processed each event
• Whether any partial failures or boundary violations were recorded

Example correlation ID lifecycle:

correlation_id: "auth-2026-05-26-001"

t=00:00.000  INGEST     POST /voice/process received
t=00:00.123  VALIDATE   SpeechResult normalized
t=00:00.131  STATE      Session reconstructed: turn_count=0, disclosure=null
t=00:00.140  POLICY     IDG-01: DISCLOSE_IDENTITY triggered (turn_count=0)
t=00:00.145  EXEC       TwiML disclosure message rendered
t=00:00.152  PERSIST    Event written — disclosure_timestamp set

6. Architecture & Scale Notes

7. Risk Register

8. One-Page Architecture Summary

[AI Voice Agent] → INGEST → VALIDATE → STATE → POLICY → EXEC → PERSIST
                                                        ↓
                                               [Event Store]
                                                        ↓
                                          [Auditor / Payer System]

Related Resources

• Developers reference → — API, traces, test suite
• Shadow Evaluation Guide → — the payer observation process
• — deterministic control walkthrough

Conformance test suite

A conformance claim you cannot verify is marketing. This suite makes the claim checkable by anyone, including people with no reason to trust the claimant.

What it is

A machine-readable specification of test cases — published as YAML in the open repository at conformance/nhid_conformance_test_suite_v1.yaml — paired with a runner that executes them against an implementation and reports pass or fail per control.

The test definitions are separate from the runner on purpose. If you write your own engine in another language, you can consume the same YAML and be measured against the same cases.

What it tests

Illustrative result

What a pass does and does not mean

Going further: Part V of the NHID-Clinical Playbook covers evidence methodology, the four separate evidence bodies, regulatory mapping and the open questions.