# Drift Detection

A mock that does not match reality is worse than no mock. aimock includes three-way drift tests that compare SDK types, real API responses, and mock output to catch shape mismatches before your users do.

## Three-Way Comparison

Each drift test compares three sources:

{ }

### SDK Types

What TypeScript types say the shape should be

⇄

### Real API

What OpenAI, Claude, Gemini actually return

⚙

### aimock

What the mock produces for the same request

#### Mock doesn't match real

aimock needs updating — test fails immediately. The SDK comparison tells us why it drifted.

#### Provider changed, SDK is behind

Early warning — the real API has new fields that neither the SDK nor aimock know about yet.

#### All three agree

No drift — the mock matches reality and the SDK types are current.

## What Drift Looks Like

$ pnpm test:drift
[critical] AIMOCK DRIFT — field in SDK + real API but missing from mock
Path: choices[].message.refusal
SDK: null Real: null Mock: <absent>
[critical] TYPE MISMATCH — real API and mock disagree on type
Path: content[].input
SDK: object Real: object Mock: string
[warning] PROVIDER ADDED FIELD — in real API but not in SDK or mock
Path: choices[].message.annotations
SDK: <absent> Real: array Mock: <absent>
✓ 2 critical (test fails) · 1 warning (logged) · detected before any user reported it

## Test Files

File | Provider | What it tests
--- | --- | ---
openai-chat.drift.ts | OpenAI | Chat Completions (streaming + non-streaming, text + tool calls)
openai-responses.drift.ts | OpenAI | Responses API (HTTP SSE)
anthropic.drift.ts | Anthropic | Claude Messages API
gemini.drift.ts | Google | Gemini generateContent + streamGenerateContent
ws-realtime.drift.ts | OpenAI | Realtime API over WebSocket
ws-responses.drift.ts | OpenAI | Responses API over WebSocket
ws-gemini-live.drift.ts | Google | Gemini Live over WebSocket
models.drift.ts | All | Model list endpoint conformance

## How Drift Analysis Works

drift-test.ts ts

    import { extractShape, triangulate, formatDriftReport, shouldFail } from "./schema";

    // 1. Get the SDK shape (what TypeScript says)
    const sdkShape = openaiChatCompletionShape();

    // 2. Call the real API and the mock in parallel
    const [realRes, mockRes] = await Promise.all([
      openaiChatNonStreaming(config, [{ role: "user", content: "Say hello" }]),
      httpPost(`${instance.url}/v1/chat/completions`, { /* ... */ }),
    ]);

    // 3. Extract response shapes
    const realShape = extractShape(realRes.body);
    const mockShape = extractShape(JSON.parse(mockRes.body));

    // 4. Three-way comparison
    const diffs = triangulate(sdkShape, realShape, mockShape);
    const report = formatDriftReport("OpenAI Chat (non-streaming text)", diffs);

    // 5. Critical diffs fail the test
    if (shouldFail(diffs)) {
      expect.soft([], report).toEqual(
        diffs.filter(d => d.severity === "critical")
      );
    }

## Severity Levels

Severity | Meaning | Action
--- | --- | ---
critical | Mock does not match real API | Test fails. aimock needs updating.
warning | Provider added new field, neither SDK nor mock have it | Logged. Early warning for future breakage.
ok | All three agree | No action needed.

## CI Integration

Drift tests run daily in CI with real API keys stored as GitHub secrets. Tests that require API keys are automatically skipped when the key is not set, so `pnpm test:drift` is safe to run locally without any keys configured.

Drift tests require real API keys and make real API calls. They are not part of the regular `pnpm test` suite and must be run explicitly with `pnpm test:drift`.

## Automatic Remediation

When drift tests detect a critical mismatch, an automated pipeline fixes, verifies, and releases the correction — no human intervention required.

### The Pipeline

1. Detect — Daily drift tests run at 6 AM UTC. On failure, a structured `drift-report.json` is produced with provider, builder file, affected functions, and field-level diffs.
2. Fix — The fix workflow triggers automatically. Claude Code receives the drift report and updates the response builders to match real API shapes, following a red-green-refactor discipline.
3. Verify — The full test suite runs (`pnpm test`) followed by drift-specific tests (`pnpm test:drift`). Both must pass.
4. PR & Merge — A PR is created with a patch version bump and changelog entry, then auto-merged once CI passes.
5. Release — On merge to main, the release workflow publishes the new version to npm, creates a GitHub Release, and notifies Slack.

If the auto-fix fails (Claude Code can't resolve the drift or tests don't pass), a GitHub issue is created with the full drift report and Claude Code logs, and a Slack alert fires for manual investigation.

### Pipeline Diagram

Drift-to-release loop text

    Daily 6 AM UTC
        │
        ▼
    ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
    │ Drift Tests  │────▶│  Claude Code  │────▶│   Verify     │
    │ (3-way diff) │     │  (auto-fix)  │     │ (test+drift) │
    └──────────────┘     └──────────────┘     └──────────────┘
        │ clean              │ fail               │ pass
        ▼                    ▼                    ▼
      (done)           GitHub Issue +       PR + auto-merge
                       Slack alert              │
                                                ▼
                                          npm publish +
                                          Slack alert

### Workflows

Workflow | File | Trigger
--- | --- | ---
Drift Tests | `test-drift.yml` | Daily cron + manual dispatch
Fix Drift | `fix-drift.yml` | Auto on drift test failure + manual dispatch
Release | `publish-release.yml` | Every push to main with unreleased version

### Slack Notifications

The pipeline posts to Slack at each stage:

- Drift detected — daily test found critical mismatches
- Drift tests passed — no drift, all clean
- Drift auto-fixed and merged — fix succeeded, PR merged
- Drift auto-fix failed — issue created, needs manual attention
- Release published — new version on npm