# Continuous Evaluation Loop Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Build on-demand dimension evaluations with golden tests and continuous production observation feedback that keep routing profiles accurate.

**Architecture:** Golden test registry (static data) → eval runner (orchestration) → eval scoring (per-dimension scorers) → profile updater (writes to ModelProvider). Production feedback hooks into existing orchestrator-evaluator to nudge dimension scores based on real conversations.

**Tech Stack:** TypeScript, Vitest, Prisma 7, PostgreSQL

**Spec:** `docs/superpowers/specs/2026-03-19-eval-loop-design.md` (EP-INF-001-P6)

---

## File Structure

### New Files

| File | Responsibility |
|---|---|
| `apps/web/lib/routing/golden-tests.ts` | Static registry of golden test prompts per dimension |
| `apps/web/lib/routing/eval-scoring.ts` | Dimension-specific scoring functions (exact match, structural, schema, tool-call, orchestrator, retrieval) |
| `apps/web/lib/routing/eval-runner.ts` | Eval orchestration — runs golden tests, computes scores, detects drift, updates profiles |
| `apps/web/lib/routing/production-feedback.ts` | Task-to-dimension mapping, score translation, two-stage accumulation/propagation |
| `apps/web/lib/routing/eval-scoring.test.ts` | Tests for scoring functions |
| `apps/web/lib/routing/eval-runner.test.ts` | Tests for eval orchestration and drift detection |
| `apps/web/lib/routing/production-feedback.test.ts` | Tests for observation feedback and guardrails |

### Modified Files

| File | Change |
|---|---|
| `apps/web/lib/orchestrator-evaluator.ts` | Add call to `updateEndpointDimensionScores` after `updatePerformanceProfile` |
| `apps/web/lib/actions/endpoint-performance.ts` | Add `triggerDimensionEval` server action |
| `apps/web/lib/routing/index.ts` | Add exports for new modules |

---

## Task 1: Golden Test Registry

**Files:**
- Create: `apps/web/lib/routing/golden-tests.ts`

- [ ] **Step 1: Create the golden test type and initial registry**

Create `apps/web/lib/routing/golden-tests.ts`:

```typescript
/**
 * EP-INF-001-P6: Golden test sets for dimension evaluation.
 * Each dimension has ~10 deterministic prompts with verifiable expected outputs.
 * These are the primary authority for capability scores.
 */
import type { BuiltinDimension } from "./types";

export type ScoringMethod =
  | "exact"        // exact string match
  | "partial"      // partial credit (correct/partial/wrong)
  | "orchestrator" // LLM-graded 1-5, scaled to 0-100
  | "structural"   // AST/pattern-based code analysis
  | "schema"       // JSON schema conformance
  | "tool_call"    // tool call structure validation
  | "retrieval";   // needle-in-haystack extraction

export interface GoldenTest {
  id: string;
  version: number;
  dimension: BuiltinDimension;
  prompt: string;
  systemPrompt?: string;
  tools?: Array<Record<string, unknown>>;
  expectedSchema?: Record<string, unknown>;
  scoring: ScoringMethod;
  expectedOutput?: string;
  maxTokens?: number;
}

// ── Reasoning Tests ──────────────────────────────────────────────────────────

const REASONING_TESTS: GoldenTest[] = [
  {
    id: "reasoning-001",
    version: 1,
    dimension: "reasoning",
    prompt: "A farmer has 17 sheep. All but 9 die. How many sheep does the farmer have left?",
    scoring: "exact",
    expectedOutput: "9",
    maxTokens: 100,
  },
  {
    id: "reasoning-002",
    version: 1,
    dimension: "reasoning",
    prompt: "If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?",
    scoring: "exact",
    expectedOutput: "5 minutes",
    maxTokens: 150,
  },
  {
    id: "reasoning-003",
    version: 1,
    dimension: "reasoning",
    prompt: "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?",
    scoring: "exact",
    expectedOutput: "$0.05",
    maxTokens: 150,
  },
  {
    id: "reasoning-004",
    version: 1,
    dimension: "reasoning",
    prompt: "Three people check into a hotel room that costs $30. They each pay $10. The manager realizes the room costs $25, so gives $5 to the bellboy to return. The bellboy keeps $2 and gives each person $1 back. Each person paid $9 (total $27), the bellboy has $2. That's $29. Where is the missing dollar?",
    scoring: "partial",
    expectedOutput: "There is no missing dollar. The $27 paid includes the $25 for the room plus the $2 the bellboy kept.",
    maxTokens: 300,
  },
  {
    id: "reasoning-005",
    version: 1,
    dimension: "reasoning",
    prompt: "In a race, you overtake the person in second place. What position are you in now?",
    scoring: "exact",
    expectedOutput: "second",
    maxTokens: 100,
  },
];

// ── Instruction Following Tests ──────────────────────────────────────────────

const INSTRUCTION_FOLLOWING_TESTS: GoldenTest[] = [
  {
    id: "instfollow-001",
    version: 1,
    dimension: "instructionFollowing",
    prompt: "List exactly 3 benefits of exercise. Use bullet points. Do not include an introduction or conclusion.",
    systemPrompt: "You must follow the user's formatting instructions exactly.",
    scoring: "partial",
    expectedOutput: "3 bullet points, no intro, no conclusion",
    maxTokens: 200,
  },
  {
    id: "instfollow-002",
    version: 1,
    dimension: "instructionFollowing",
    prompt: "Respond with exactly the word 'CONFIRMED' and nothing else.",
    scoring: "exact",
    expectedOutput: "CONFIRMED",
    maxTokens: 10,
  },
  {
    id: "instfollow-003",
    version: 1,
    dimension: "instructionFollowing",
    prompt: "Write a haiku about databases. It must have exactly 3 lines with 5, 7, and 5 syllables respectively.",
    scoring: "partial",
    expectedOutput: "3 lines, approximately 5-7-5 syllables",
    maxTokens: 100,
  },
  {
    id: "instfollow-004",
    version: 1,
    dimension: "instructionFollowing",
    prompt: "Answer in JSON format only: What are the three primary colors? Use the key 'colors' with an array value.",
    scoring: "schema",
    expectedSchema: {
      type: "object",
      properties: { colors: { type: "array", items: { type: "string" } } },
      required: ["colors"],
    },
    maxTokens: 100,
  },
  {
    id: "instfollow-005",
    version: 1,
    dimension: "instructionFollowing",
    prompt: "Respond in exactly 2 sentences. The first sentence must start with 'Yes' and the second with 'However'. Topic: remote work.",
    scoring: "partial",
    expectedOutput: "2 sentences, first starts with Yes, second starts with However",
    maxTokens: 150,
  },
];

// ── Tool Fidelity Tests ──────────────────────────────────────────────────────

const TOOL_FIDELITY_TESTS: GoldenTest[] = [
  {
    id: "toolfidelity-001",
    version: 1,
    dimension: "toolFidelity",
    prompt: "Create a new backlog item titled 'Fix login timeout' with status 'open'.",
    tools: [
      {
        type: "function",
        function: {
          name: "create_backlog_item",
          description: "Create a new backlog item",
          parameters: {
            type: "object",
            properties: {
              title: { type: "string" },
              description: { type: "string" },
              status: { type: "string", enum: ["open", "in-progress", "done"] },
            },
            required: ["title", "status"],
          },
        },
      },
    ],
    scoring: "tool_call",
    expectedOutput: "create_backlog_item",
    maxTokens: 200,
  },
  {
    id: "toolfidelity-002",
    version: 1,
    dimension: "toolFidelity",
    prompt: "What is the weather like today?",
    tools: [
      {
        type: "function",
        function: {
          name: "create_backlog_item",
          description: "Create a new backlog item",
          parameters: {
            type: "object",
            properties: { title: { type: "string" }, status: { type: "string" } },
            required: ["title", "status"],
          },
        },
      },
    ],
    scoring: "tool_call",
    expectedOutput: "__ABSTAIN__",
    maxTokens: 200,
  },
  {
    id: "toolfidelity-003",
    version: 1,
    dimension: "toolFidelity",
    prompt: "Search the web for 'TypeScript 5.5 release notes' and summarize the key features.",
    tools: [
      {
        type: "function",
        function: {
          name: "web_search",
          description: "Search the web for information",
          parameters: {
            type: "object",
            properties: { query: { type: "string" } },
            required: ["query"],
          },
        },
      },
      {
        type: "function",
        function: {
          name: "create_backlog_item",
          description: "Create a new backlog item",
          parameters: {
            type: "object",
            properties: { title: { type: "string" } },
            required: ["title"],
          },
        },
      },
    ],
    scoring: "tool_call",
    expectedOutput: "web_search",
    maxTokens: 200,
  },
];

// ── Structured Output Tests ──────────────────────────────────────────────────

const STRUCTURED_OUTPUT_TESTS: GoldenTest[] = [
  {
    id: "structout-001",
    version: 1,
    dimension: "structuredOutput",
    prompt: "Return a JSON object with fields: name (string), age (number), active (boolean). Use realistic values.",
    scoring: "schema",
    expectedSchema: {
      type: "object",
      properties: {
        name: { type: "string" },
        age: { type: "number" },
        active: { type: "boolean" },
      },
      required: ["name", "age", "active"],
    },
    maxTokens: 100,
  },
  {
    id: "structout-002",
    version: 1,
    dimension: "structuredOutput",
    prompt: "Return a JSON array of exactly 3 objects, each with 'id' (number) and 'label' (string) fields.",
    scoring: "schema",
    expectedSchema: {
      type: "array",
      items: {
        type: "object",
        properties: { id: { type: "number" }, label: { type: "string" } },
        required: ["id", "label"],
      },
      minItems: 3,
      maxItems: 3,
    },
    maxTokens: 200,
  },
  {
    id: "structout-003",
    version: 1,
    dimension: "structuredOutput",
    systemPrompt: "Always respond in valid JSON. No markdown, no explanation.",
    prompt: "Classify the sentiment of 'I love this product!' as positive, negative, or neutral. Return {\"text\": \"...\", \"sentiment\": \"...\"}",
    scoring: "schema",
    expectedSchema: {
      type: "object",
      properties: {
        text: { type: "string" },
        sentiment: { type: "string", enum: ["positive", "negative", "neutral"] },
      },
      required: ["text", "sentiment"],
    },
    maxTokens: 100,
  },
];

// ── Conversational Tests ─────────────────────────────────────────────────────

const CONVERSATIONAL_TESTS: GoldenTest[] = [
  {
    id: "convo-001",
    version: 1,
    dimension: "conversational",
    prompt: "Hi! How are you today?",
    scoring: "orchestrator",
    maxTokens: 200,
  },
  {
    id: "convo-002",
    version: 1,
    dimension: "conversational",
    prompt: "Can you explain what you can help me with in simple terms? I'm not very technical.",
    scoring: "orchestrator",
    maxTokens: 300,
  },
  {
    id: "convo-003",
    version: 1,
    dimension: "conversational",
    prompt: "I just said I don't want to use that approach. Why are you suggesting it again?",
    systemPrompt: "The user previously rejected using microservices. You suggested microservices anyway.",
    scoring: "orchestrator",
    maxTokens: 200,
  },
];

// ── Codegen Tests ────────────────────────────────────────────────────────────

const CODEGEN_TESTS: GoldenTest[] = [
  {
    id: "codegen-001",
    version: 1,
    dimension: "codegen",
    prompt: "Write a TypeScript function called `isPalindrome` that takes a string and returns true if it's a palindrome (case-insensitive, ignoring spaces and punctuation).",
    scoring: "structural",
    expectedOutput: "function isPalindrome",
    maxTokens: 500,
  },
  {
    id: "codegen-002",
    version: 1,
    dimension: "codegen",
    prompt: "Write a TypeScript function called `groupBy` that takes an array of objects and a key name, and returns an object where each key is a unique value of that property and the value is an array of matching objects.",
    scoring: "structural",
    expectedOutput: "function groupBy",
    maxTokens: 500,
  },
  {
    id: "codegen-003",
    version: 1,
    dimension: "codegen",
    prompt: "Write a TypeScript function called `debounce` that takes a function and a delay in milliseconds, and returns a debounced version.",
    scoring: "structural",
    expectedOutput: "function debounce",
    maxTokens: 500,
  },
];

// ── Context Retention Tests ──────────────────────────────────────────────────

const CONTEXT_RETENTION_TESTS: GoldenTest[] = [
  {
    id: "context-001",
    version: 1,
    dimension: "contextRetention",
    systemPrompt: "You are helping a user with project planning. The user previously told you: 'Our project codename is Phoenix, we're targeting Q3 launch, and the budget is $2.5M.' Remember all details they've shared.",
    prompt: "What was the project codename I mentioned?",
    scoring: "retrieval",
    expectedOutput: "Phoenix",
    maxTokens: 50,
  },
  {
    id: "context-002",
    version: 1,
    dimension: "contextRetention",
    systemPrompt: "You are a technical consultant. The user has shared the following context across your conversation: 'My team has 7 developers. We primarily use Rust for our backend. Our main database is PostgreSQL. We deploy to AWS us-east-1. Our CI runs on GitHub Actions.' Answer questions about what the user has told you.",
    prompt: "How many developers are on my team and what language do we use?",
    scoring: "retrieval",
    expectedOutput: "7 developers, Rust",
    maxTokens: 100,
  },
];

// ── Public Registry ──────────────────────────────────────────────────────────

export const GOLDEN_TESTS: GoldenTest[] = [
  ...REASONING_TESTS,
  ...INSTRUCTION_FOLLOWING_TESTS,
  ...TOOL_FIDELITY_TESTS,
  ...STRUCTURED_OUTPUT_TESTS,
  ...CONVERSATIONAL_TESTS,
  ...CODEGEN_TESTS,
  ...CONTEXT_RETENTION_TESTS,
];

/** Get golden tests for a specific dimension. */
export function getTestsForDimension(dimension: BuiltinDimension): GoldenTest[] {
  return GOLDEN_TESTS.filter((t) => t.dimension === dimension);
}
```

- [ ] **Step 2: Commit**

```bash
git add apps/web/lib/routing/golden-tests.ts
git commit -m "feat: add golden test registry for 7 capability dimensions (EP-INF-001-P6)"
```

---

## Task 2: Eval Scoring Functions — TDD

**Files:**
- Create: `apps/web/lib/routing/eval-scoring.test.ts`
- Create: `apps/web/lib/routing/eval-scoring.ts`

- [ ] **Step 1: Write failing tests for scoring functions**

Create `apps/web/lib/routing/eval-scoring.test.ts`:

```typescript
import { describe, expect, it } from "vitest";
import {
  scoreExact,
  scorePartial,
  scoreSchema,
  scoreToolCall,
  scoreStructural,
  scoreRetrieval,
  scoreDimension,
} from "./eval-scoring";
import type { GoldenTest } from "./golden-tests";

describe("scoreExact", () => {
  it("returns 10 for exact match (case-insensitive, trimmed)", () => {
    expect(scoreExact("  9  ", "9")).toBe(10);
  });
  it("returns 10 when expected string is contained in response", () => {
    expect(scoreExact("The answer is 9 sheep.", "9")).toBe(10);
  });
  it("returns 0 for no match", () => {
    expect(scoreExact("The answer is 8", "9")).toBe(0);
  });
});

describe("scorePartial", () => {
  it("returns 10 for response containing key expected content", () => {
    expect(scorePartial(
      "There is no missing dollar. The $27 includes the room and bellboy tip.",
      "There is no missing dollar",
    )).toBe(10);
  });
  it("returns 5 for partial match (shares significant keywords)", () => {
    expect(scorePartial(
      "There was no dollar that went missing from the total.",
      "There is no missing dollar",
    )).toBe(5);
  });
  it("returns 0 for completely wrong answer with no keyword overlap", () => {
    expect(scorePartial("The bellboy kept the change for himself.", "There is no missing dollar")).toBe(0);
  });
});

describe("scoreSchema", () => {
  it("returns 10 for valid JSON matching schema", () => {
    const schema = {
      type: "object",
      properties: { name: { type: "string" }, age: { type: "number" } },
      required: ["name", "age"],
    };
    expect(scoreSchema('{"name": "Alice", "age": 30}', schema)).toBe(10);
  });
  it("returns 0 for invalid JSON", () => {
    expect(scoreSchema("not json", { type: "object" })).toBe(0);
  });
  it("returns 5 for valid JSON missing required field", () => {
    const schema = {
      type: "object",
      properties: { name: { type: "string" }, age: { type: "number" } },
      required: ["name", "age"],
    };
    expect(scoreSchema('{"name": "Alice"}', schema)).toBe(5);
  });
});

describe("scoreToolCall", () => {
  it("returns 10 for correct tool called", () => {
    const toolCalls = [{ name: "create_backlog_item", arguments: { title: "Fix login", status: "open" } }];
    expect(scoreToolCall(toolCalls, "create_backlog_item")).toBe(10);
  });
  it("returns 10 for correct abstention (no tool call when __ABSTAIN__ expected)", () => {
    expect(scoreToolCall([], "__ABSTAIN__")).toBe(10);
  });
  it("returns 0 for wrong tool called", () => {
    const toolCalls = [{ name: "web_search", arguments: { query: "test" } }];
    expect(scoreToolCall(toolCalls, "create_backlog_item")).toBe(0);
  });
  it("returns 0 for tool called when abstention expected", () => {
    const toolCalls = [{ name: "create_backlog_item", arguments: {} }];
    expect(scoreToolCall(toolCalls, "__ABSTAIN__")).toBe(0);
  });
});

describe("scoreStructural", () => {
  it("returns 10 for response containing expected function definition", () => {
    const code = "function isPalindrome(str: string): boolean { return str === str.split('').reverse().join(''); }";
    expect(scoreStructural(code, "function isPalindrome")).toBe(10);
  });
  it("returns 5 for response with code but wrong function name", () => {
    const code = "function checkPalindrome(str: string): boolean { return true; }";
    expect(scoreStructural(code, "function isPalindrome")).toBe(5);
  });
  it("returns 0 for no code at all", () => {
    expect(scoreStructural("I can help you write that function.", "function isPalindrome")).toBe(0);
  });
});

describe("scoreRetrieval", () => {
  it("returns 10 for exact retrieval", () => {
    expect(scoreRetrieval("The project codename is Phoenix.", "Phoenix")).toBe(10);
  });
  it("returns 0 for wrong retrieval", () => {
    expect(scoreRetrieval("The project codename is Falcon.", "Phoenix")).toBe(0);
  });
});

describe("scoreDimension", () => {
  it("normalizes per-test scores to 0-100", () => {
    const scores = [10, 10, 10, 0, 0]; // 30/50 = 60%
    expect(scoreDimension(scores)).toBe(60);
  });
  it("returns 0 for all failures", () => {
    expect(scoreDimension([0, 0, 0])).toBe(0);
  });
  it("returns 100 for all perfect", () => {
    expect(scoreDimension([10, 10, 10])).toBe(100);
  });
});
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `cd apps/web && npx vitest run lib/routing/eval-scoring.test.ts`

- [ ] **Step 3: Implement eval-scoring.ts**

Create `apps/web/lib/routing/eval-scoring.ts`:

```typescript
/**
 * EP-INF-001-P6: Dimension-specific scoring functions for golden tests.
 * Each function scores a single response on a 0-10 scale.
 * scoreDimension normalizes an array of per-test scores to 0-100.
 */

/** Exact match: response contains the expected string (case-insensitive). */
export function scoreExact(response: string, expected: string): number {
  const normalized = response.trim().toLowerCase();
  const target = expected.trim().toLowerCase();
  return normalized.includes(target) ? 10 : 0;
}

/** Partial credit: 10 for key content present, 5 for related but imprecise, 0 for wrong. */
export function scorePartial(response: string, expected: string): number {
  const normalized = response.toLowerCase();
  const target = expected.toLowerCase();
  // Full match — key content present
  if (normalized.includes(target)) return 10;
  // Partial — shares significant keywords (>50% of expected words found)
  const expectedWords = target.split(/\s+/).filter((w) => w.length > 3);
  const matchCount = expectedWords.filter((w) => normalized.includes(w)).length;
  if (expectedWords.length > 0 && matchCount / expectedWords.length > 0.5) return 5;
  return 0;
}

/** Schema validation: 10 for valid JSON matching schema, 5 for valid JSON partial match, 0 for invalid. */
export function scoreSchema(
  response: string,
  schema: Record<string, unknown>,
): number {
  // Extract JSON from response (may be wrapped in markdown code fences)
  const jsonMatch = response.match(/```(?:json)?\s*([\s\S]*?)```/) ??
    response.match(/(\{[\s\S]*\})/) ??
    response.match(/(\[[\s\S]*\])/);
  const jsonStr = jsonMatch ? jsonMatch[1]!.trim() : response.trim();

  let parsed: unknown;
  try {
    parsed = JSON.parse(jsonStr);
  } catch {
    return 0;
  }

  // Basic type check
  const expectedType = schema.type as string;
  if (expectedType === "object" && (typeof parsed !== "object" || Array.isArray(parsed) || parsed === null)) return 5;
  if (expectedType === "array" && !Array.isArray(parsed)) return 5;

  // Required fields check
  const required = (schema.required as string[]) ?? [];
  if (expectedType === "object" && typeof parsed === "object" && parsed !== null) {
    const keys = Object.keys(parsed as Record<string, unknown>);
    const missingRequired = required.filter((r) => !keys.includes(r));
    if (missingRequired.length > 0) return 5;
  }

  // Array length check
  if (expectedType === "array" && Array.isArray(parsed)) {
    const minItems = schema.minItems as number | undefined;
    const maxItems = schema.maxItems as number | undefined;
    if (minItems !== undefined && parsed.length < minItems) return 5;
    if (maxItems !== undefined && parsed.length > maxItems) return 5;
  }

  return 10;
}

/** Tool call validation: correct tool called, correct abstention. */
export function scoreToolCall(
  toolCalls: Array<{ name: string; arguments: Record<string, unknown> }>,
  expected: string,
): number {
  if (expected === "__ABSTAIN__") {
    return toolCalls.length === 0 ? 10 : 0;
  }
  const called = toolCalls.find((tc) => tc.name === expected);
  return called ? 10 : 0;
}

/** Structural code analysis: checks for expected function definition patterns. */
export function scoreStructural(response: string, expectedPattern: string): number {
  // Extract code from markdown fences if present
  const codeMatch = response.match(/```(?:\w+)?\s*([\s\S]*?)```/);
  const code = codeMatch ? codeMatch[1]!.trim() : response;

  // Check for the expected pattern (e.g., "function isPalindrome")
  if (code.includes(expectedPattern)) return 10;

  // Check if response contains any function/code at all
  const hasCode = /(?:function\s+\w+|const\s+\w+\s*=|=>\s*\{|class\s+\w+)/.test(code);
  return hasCode ? 5 : 0;
}

/** Retrieval accuracy: expected value found in response. */
export function scoreRetrieval(response: string, expected: string): number {
  return response.toLowerCase().includes(expected.toLowerCase()) ? 10 : 0;
}

/** Normalize an array of per-test scores (each 0-10) to a single 0-100 dimension score. */
export function scoreDimension(scores: number[]): number {
  if (scores.length === 0) return 0;
  const total = scores.reduce((a, b) => a + b, 0);
  const maxPossible = scores.length * 10;
  return Math.round((total / maxPossible) * 100);
}
```

- [ ] **Step 4: Run tests to verify they pass**

Run: `cd apps/web && npx vitest run lib/routing/eval-scoring.test.ts`

- [ ] **Step 5: Commit**

```bash
git add apps/web/lib/routing/eval-scoring.ts apps/web/lib/routing/eval-scoring.test.ts
git commit -m "feat: implement eval scoring functions with TDD (EP-INF-001-P6)"
```

---

## Task 3: Eval Runner — TDD

**Files:**
- Create: `apps/web/lib/routing/eval-runner.test.ts`
- Create: `apps/web/lib/routing/eval-runner.ts`

- [ ] **Step 1: Write failing tests**

Create `apps/web/lib/routing/eval-runner.test.ts`:

```typescript
import { describe, expect, it } from "vitest";
import {
  computeNewScore,
  detectDrift,
  type DriftResult,
} from "./eval-runner";

describe("computeNewScore", () => {
  it("uses raw score on first eval (evalCount=0)", () => {
    expect(computeNewScore(85, 50, 0)).toBe(85);
  });
  it("uses weighted rolling average on subsequent evals", () => {
    // 0.7 * 85 + 0.3 * 70 = 59.5 + 21 = 80.5 → 81
    expect(computeNewScore(85, 70, 5)).toBe(81);
  });
  it("clamps to 0-100 range", () => {
    expect(computeNewScore(150, 90, 3)).toBeLessThanOrEqual(100);
    expect(computeNewScore(-10, 20, 3)).toBeGreaterThanOrEqual(0);
  });
});

describe("detectDrift", () => {
  it("returns no drift for small changes", () => {
    const result = detectDrift(80, 70);
    expect(result.severity).toBe("none");
  });
  it("returns warning for >15 point drop", () => {
    const result = detectDrift(55, 80);
    expect(result.severity).toBe("warning");
  });
  it("returns severe for >25 point drop", () => {
    const result = detectDrift(50, 80);
    expect(result.severity).toBe("severe");
  });
  it("returns no drift for improvements", () => {
    const result = detectDrift(90, 70);
    expect(result.severity).toBe("none");
  });
});
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `cd apps/web && npx vitest run lib/routing/eval-runner.test.ts`

- [ ] **Step 3: Implement eval-runner.ts**

Create `apps/web/lib/routing/eval-runner.ts`:

```typescript
/**
 * EP-INF-001-P6: Eval runner — orchestrates golden test evaluation,
 * computes dimension scores, detects drift, updates ModelProvider profiles.
 */
import { prisma } from "@dpf/db";
import * as crypto from "crypto";
import { callProvider } from "@/lib/ai-inference";
import type { BuiltinDimension } from "./types";
import { BUILTIN_DIMENSIONS } from "./types";
import { getTestsForDimension, type GoldenTest, type ScoringMethod } from "./golden-tests";
import {
  scoreExact,
  scorePartial,
  scoreSchema,
  scoreToolCall,
  scoreStructural,
  scoreRetrieval,
  scoreDimension,
} from "./eval-scoring";

// ── Score Computation ────────────────────────────────────────────────────────

/** Compute new dimension score from eval result and previous score. */
export function computeNewScore(
  evalScore: number,
  previousScore: number,
  evalCount: number,
): number {
  const raw = evalCount === 0
    ? evalScore
    : 0.7 * evalScore + 0.3 * previousScore;
  return Math.round(Math.min(100, Math.max(0, raw)));
}

// ── Drift Detection ──────────────────────────────────────────────────────────

export type DriftResult = {
  severity: "none" | "warning" | "severe";
  delta: number;
};

/** Compare raw eval score against previous stored score. */
export function detectDrift(rawEvalScore: number, previousScore: number): DriftResult {
  const delta = previousScore - rawEvalScore;
  if (delta > 25) return { severity: "severe", delta };
  if (delta > 15) return { severity: "warning", delta };
  return { severity: "none", delta };
}

// ── Test Execution ───────────────────────────────────────────────────────────

interface TestResult {
  testId: string;
  version: number;
  scoring: ScoringMethod;
  score: number;       // 0-10
  response: string;
  error?: string;
}

/** Score a single response using the appropriate scoring method. */
function scoreResponse(
  test: GoldenTest,
  content: string,
  toolCalls: Array<{ name: string; arguments: Record<string, unknown> }>,
): number {
  switch (test.scoring) {
    case "exact":
      return scoreExact(content, test.expectedOutput ?? "");
    case "partial":
      return scorePartial(content, test.expectedOutput ?? "");
    case "schema":
      return scoreSchema(content, test.expectedSchema ?? {});
    case "tool_call":
      return scoreToolCall(toolCalls, test.expectedOutput ?? "");
    case "structural":
      return scoreStructural(content, test.expectedOutput ?? "");
    case "retrieval":
      return scoreRetrieval(content, test.expectedOutput ?? "");
    case "orchestrator":
      // KNOWN GAP: Orchestrator scoring requires a separate grading call to a different
      // endpoint (to avoid self-evaluation). This is not implemented in Phase 6 — the
      // conversational dimension will score neutral (5/10 = 50/100) until orchestrator
      // grading is added. This is acceptable because:
      // 1. Conversational quality is the least routing-critical dimension
      // 2. Production observations will nudge the score over time
      // 3. The infrastructure for cross-endpoint grading needs the eval runner itself to be stable first
      return 5;
    default:
      return 0;
  }
}

/** Run a single golden test against an endpoint. */
async function runGoldenTest(
  endpointId: string,
  modelId: string,
  test: GoldenTest,
): Promise<TestResult> {
  try {
    const messages = [{ role: "user" as const, content: test.prompt }];
    const result = await callProvider(
      endpointId,
      modelId,
      messages,
      test.systemPrompt ?? "You are a helpful assistant.",
      test.tools,
    );

    const score = scoreResponse(test, result.content, result.toolCalls ?? []);
    return {
      testId: test.id,
      version: test.version,
      scoring: test.scoring,
      score,
      response: result.content.slice(0, 500),
    };
  } catch (e) {
    return {
      testId: test.id,
      version: test.version,
      scoring: test.scoring,
      score: 0,
      response: "",
      error: e instanceof Error ? e.message : String(e),
    };
  }
}

// ── Dimension Eval Orchestration ─────────────────────────────────────────────

interface DimensionEvalResult {
  dimension: BuiltinDimension;
  rawScore: number;         // 0-100 from this eval
  newScore: number;         // after rolling average
  previousScore: number;
  drift: DriftResult;
  testResults: TestResult[];
  inconclusive: boolean;    // >50% of tests failed to run
}

/** Resolve the best modelId for a provider (same as fallback.ts). */
async function resolveModelId(providerId: string): Promise<string> {
  const profile = await prisma.modelProfile.findFirst({
    where: { providerId },
    orderBy: { generatedAt: "desc" },
    select: { modelId: true },
  });
  if (profile) return profile.modelId;

  const discovered = await prisma.discoveredModel.findFirst({
    where: { providerId, NOT: { modelId: { contains: "embed" } } },
    orderBy: { modelId: "asc" },
    select: { modelId: true },
  });
  return discovered?.modelId ?? "";
}

/** Run golden test evaluation for one dimension on one endpoint. */
async function evalDimension(
  endpointId: string,
  modelId: string,
  dimension: BuiltinDimension,
  previousScore: number,
  evalCount: number,
): Promise<DimensionEvalResult> {
  const tests = getTestsForDimension(dimension);
  const testResults: TestResult[] = [];

  for (const test of tests) {
    const result = await runGoldenTest(endpointId, modelId, test);
    testResults.push(result);
  }

  // Check if inconclusive (>50% errors)
  const errorCount = testResults.filter((r) => r.error).length;
  const inconclusive = errorCount > tests.length / 2;

  if (inconclusive) {
    return {
      dimension,
      rawScore: previousScore,
      newScore: previousScore,
      previousScore,
      drift: { severity: "none", delta: 0 },
      testResults,
      inconclusive: true,
    };
  }

  const rawScore = scoreDimension(testResults.map((r) => r.score));
  const drift = detectDrift(rawScore, previousScore);
  const newScore = computeNewScore(rawScore, previousScore, evalCount);

  return {
    dimension,
    rawScore,
    newScore,
    previousScore,
    drift,
    testResults,
    inconclusive: false,
  };
}

// ── Public API ───────────────────────────────────────────────────────────────

export interface EvalRunResult {
  endpointId: string;
  modelId: string;
  dimensions: DimensionEvalResult[];
  testRunId: string;
  hasDrift: boolean;
  hasSevereDrift: boolean;
}

/**
 * Run a full dimension evaluation for one endpoint.
 * Updates ModelProvider capability scores and creates an EndpointTestRun record.
 */
export async function runDimensionEval(
  endpointId: string,
  triggeredBy: string,
): Promise<EvalRunResult> {
  const modelId = await resolveModelId(endpointId);
  const provider = await prisma.modelProvider.findUnique({
    where: { providerId: endpointId },
  });

  if (!provider) throw new Error(`Provider ${endpointId} not found`);

  const runId = `DE-${crypto.randomUUID().slice(0, 8).toUpperCase()}`;

  // Create the test run record
  await prisma.endpointTestRun.create({
    data: {
      runId,
      endpointId,
      modelId,
      taskType: "dimension-eval",
      triggeredBy,
      status: "running",
    },
  });

  // Evaluate each dimension
  const dimensions: DimensionEvalResult[] = [];
  for (const dim of BUILTIN_DIMENSIONS) {
    const previousScore = provider[dim] as number;
    const result = await evalDimension(endpointId, modelId, dim, previousScore, provider.evalCount);
    dimensions.push(result);
  }

  // Update ModelProvider with new scores (skip inconclusive dimensions)
  const scoreUpdates: Record<string, number> = {};
  for (const d of dimensions) {
    if (!d.inconclusive) {
      scoreUpdates[d.dimension] = d.newScore;
    }
  }

  const hasDrift = dimensions.some((d) => d.drift.severity !== "none");
  const hasSevereDrift = dimensions.some((d) => d.drift.severity === "severe");

  await prisma.modelProvider.update({
    where: { providerId: endpointId },
    data: {
      ...scoreUpdates,
      profileSource: "evaluated",
      profileConfidence: (provider.evalCount + 1) >= 5 ? "high" : "medium",
      evalCount: { increment: 1 },
      lastEvalAt: new Date(),
      // Mark degraded on severe drift
      ...(hasSevereDrift ? { status: "degraded" } : {}),
    },
  });

  // Complete the test run record
  await prisma.endpointTestRun.update({
    where: { runId },
    data: {
      status: "completed",
      completedAt: new Date(),
      avgScore: dimensions.reduce((a, d) => a + d.rawScore, 0) / dimensions.length,
      results: {
        dimensions: dimensions.map((d) => ({
          dimension: d.dimension,
          rawScore: d.rawScore,
          newScore: d.newScore,
          previousScore: d.previousScore,
          drift: d.drift,
          inconclusive: d.inconclusive,
          tests: d.testResults,
        })),
      },
    },
  });

  return {
    endpointId,
    modelId,
    dimensions,
    testRunId: runId,
    hasDrift,
    hasSevereDrift,
  };
}

/**
 * Run dimension evaluation for ALL active endpoints.
 */
export async function runAllDimensionEvals(triggeredBy: string): Promise<EvalRunResult[]> {
  const providers = await prisma.modelProvider.findMany({
    where: { status: { in: ["active", "degraded"] }, endpointType: "llm", retiredAt: null },
    select: { providerId: true },
  });

  const results: EvalRunResult[] = [];
  for (const p of providers) {
    try {
      const result = await runDimensionEval(p.providerId, triggeredBy);
      results.push(result);
    } catch (e) {
      console.error(`[eval-runner] failed to evaluate ${p.providerId}:`, e);
    }
  }
  return results;
}
```

- [ ] **Step 4: Run tests to verify they pass**

Run: `cd apps/web && npx vitest run lib/routing/eval-runner.test.ts`

- [ ] **Step 5: Update barrel export**

Add to `apps/web/lib/routing/index.ts`:

```typescript
export { runDimensionEval, runAllDimensionEvals, computeNewScore, detectDrift } from "./eval-runner";
export type { EvalRunResult } from "./eval-runner";
export { scoreDimension } from "./eval-scoring";
export { GOLDEN_TESTS, getTestsForDimension } from "./golden-tests";
```

- [ ] **Step 6: Commit**

```bash
git add apps/web/lib/routing/
git commit -m "feat: implement eval runner with golden tests and drift detection (EP-INF-001-P6)"
```

---

## Task 4: Production Observation Feedback — TDD

**Files:**
- Create: `apps/web/lib/routing/production-feedback.test.ts`
- Create: `apps/web/lib/routing/production-feedback.ts`

- [ ] **Step 1: Write failing tests**

Create `apps/web/lib/routing/production-feedback.test.ts`:

```typescript
import { describe, expect, it } from "vitest";
import {
  getDimensionsForTask,
  computeObservationDelta,
} from "./production-feedback";

describe("getDimensionsForTask", () => {
  it("returns reasoning for reasoning task", () => {
    const dims = getDimensionsForTask("reasoning");
    expect(dims).toEqual([{ dimension: "reasoning", weight: 1.0 }]);
  });
  it("returns primary + secondary for code-gen", () => {
    const dims = getDimensionsForTask("code-gen");
    expect(dims).toEqual([
      { dimension: "codegen", weight: 1.0 },
      { dimension: "instructionFollowing", weight: 0.5 },
    ]);
  });
  it("returns empty for unknown task type", () => {
    expect(getDimensionsForTask("unknown")).toEqual([]);
  });
  it("returns empty for undefined", () => {
    expect(getDimensionsForTask(undefined as unknown as string)).toEqual([]);
  });
});

describe("computeObservationDelta", () => {
  it("returns +8 for orchestrator score 5", () => {
    expect(computeObservationDelta(5)).toBe(8);
  });
  it("returns 0 for score 3 (neutral)", () => {
    expect(computeObservationDelta(3)).toBe(0);
  });
  it("returns -8 for score 1", () => {
    expect(computeObservationDelta(1)).toBe(-8);
  });
  it("returns -4 for score 2", () => {
    expect(computeObservationDelta(2)).toBe(-4);
  });
});
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `cd apps/web && npx vitest run lib/routing/production-feedback.test.ts`

- [ ] **Step 3: Implement production-feedback.ts**

Create `apps/web/lib/routing/production-feedback.ts`:

```typescript
/**
 * EP-INF-001-P6: Production observation feedback.
 * Maps orchestrator scores from real conversations to dimension nudges.
 * Two-stage: accumulate in EndpointTaskPerformance.dimensionScores,
 * propagate to ModelProvider once threshold is met.
 */
import { prisma } from "@dpf/db";
import type { BuiltinDimension } from "./types";

// ── Task-to-Dimension Mapping ────────────────────────────────────────────────

interface DimensionMapping {
  dimension: BuiltinDimension;
  weight: number; // 1.0 for primary, 0.5 for secondary
}

const TASK_DIMENSION_MAP: Record<string, DimensionMapping[]> = {
  "reasoning":       [{ dimension: "reasoning", weight: 1.0 }],
  "code-gen":        [{ dimension: "codegen", weight: 1.0 }, { dimension: "instructionFollowing", weight: 0.5 }],
  "tool-action":     [{ dimension: "toolFidelity", weight: 1.0 }],
  "data-extraction": [{ dimension: "structuredOutput", weight: 1.0 }],
  "summarization":   [{ dimension: "instructionFollowing", weight: 1.0 }],
  "greeting":        [{ dimension: "conversational", weight: 1.0 }],
  "creative":        [{ dimension: "conversational", weight: 1.0 }, { dimension: "reasoning", weight: 0.5 }],
  "web-search":      [{ dimension: "toolFidelity", weight: 1.0 }],
  "status-query":    [{ dimension: "instructionFollowing", weight: 1.0 }],
};

/** Get the dimension mappings for a task type. Returns empty for unknown. */
export function getDimensionsForTask(taskType: string): DimensionMapping[] {
  if (!taskType) return [];
  return TASK_DIMENSION_MAP[taskType] ?? [];
}

/** Compute the dimension delta from an orchestrator score (1-5). */
export function computeObservationDelta(orchestratorScore: number): number {
  return (orchestratorScore - 3) * 4;
}

// ── Accumulation & Propagation ───────────────────────────────────────────────

const PROPAGATION_THRESHOLD = 5;

interface DimensionTally {
  count: number;
  totalDelta: number;
}

/**
 * Update endpoint dimension scores based on a production observation.
 * Called from orchestrator-evaluator after every conversation scoring.
 *
 * Two-stage flow:
 * 1. Accumulate per-task-type deltas in EndpointTaskPerformance.dimensionScores
 * 2. When a dimension's total count across all task types reaches threshold,
 *    propagate the average delta to ModelProvider
 */
export async function updateEndpointDimensionScores(
  endpointId: string,
  taskType: string,
  orchestratorScore: number,
): Promise<void> {
  const mappings = getDimensionsForTask(taskType);
  if (mappings.length === 0) return;

  const baseDelta = computeObservationDelta(orchestratorScore);

  // Check golden test recency — skip if endpoint was evaluated within 24 hours.
  // This is endpoint-wide (not per-dimension) — a deliberate simplification using ModelProvider.lastEvalAt.
  const provider = await prisma.modelProvider.findUnique({
    where: { providerId: endpointId },
    select: { lastEvalAt: true },
  });
  if (provider?.lastEvalAt) {
    const hoursSinceEval = (Date.now() - provider.lastEvalAt.getTime()) / (1000 * 60 * 60);
    if (hoursSinceEval < 24) return;
  }

  // Get or create the performance record for this task type
  const perf = await prisma.endpointTaskPerformance.findUnique({
    where: { endpointId_taskType: { endpointId, taskType } },
  });

  const currentDimScores = (perf?.dimensionScores as Record<string, DimensionTally>) ?? {};

  // Accumulate deltas
  for (const { dimension, weight } of mappings) {
    const delta = Math.round(baseDelta * weight);
    const tally = currentDimScores[dimension] ?? { count: 0, totalDelta: 0 };
    tally.count += 1;
    tally.totalDelta += delta;
    currentDimScores[dimension] = tally;
  }

  // Upsert EndpointTaskPerformance (creates record if first observation for this task type)
  await prisma.endpointTaskPerformance.upsert({
    where: { endpointId_taskType: { endpointId, taskType } },
    update: { dimensionScores: currentDimScores },
    create: {
      endpointId,
      taskType,
      dimensionScores: currentDimScores,
    },
  });

  // Check propagation threshold across ALL task types for this endpoint
  const allPerfs = await prisma.endpointTaskPerformance.findMany({
    where: { endpointId },
    select: { id: true, dimensionScores: true },
  });

  // Aggregate tallies across all task types
  const aggregated: Record<string, DimensionTally> = {};
  for (const p of allPerfs) {
    const scores = (p.dimensionScores as Record<string, DimensionTally>) ?? {};
    for (const [dim, tally] of Object.entries(scores)) {
      if (!aggregated[dim]) aggregated[dim] = { count: 0, totalDelta: 0 };
      aggregated[dim].count += tally.count;
      aggregated[dim].totalDelta += tally.totalDelta;
    }
  }

  // Propagate dimensions that have reached the threshold
  const updates: Record<string, number> = {};
  const resetDimensions: string[] = [];

  const providerFull = await prisma.modelProvider.findUnique({
    where: { providerId: endpointId },
  });
  if (!providerFull) return;

  // Only propagate if profileSource is "seed" — don't downgrade "evaluated"
  const canPropagate = providerFull.profileSource === "seed";

  for (const [dim, tally] of Object.entries(aggregated)) {
    if (tally.count >= PROPAGATION_THRESHOLD && canPropagate) {
      const avgDelta = Math.round(tally.totalDelta / tally.count);
      const currentScore = (providerFull as Record<string, unknown>)[dim] as number ?? 50;
      updates[dim] = Math.min(100, Math.max(0, currentScore + avgDelta));
      resetDimensions.push(dim);
    }
  }

  if (Object.keys(updates).length > 0) {
    await prisma.modelProvider.update({
      where: { providerId: endpointId },
      data: {
        ...updates,
        profileSource: "production",
      },
    });

    // Reset tallies for propagated dimensions across all task types
    for (const p of allPerfs) {
      const scores = (p.dimensionScores as Record<string, DimensionTally>) ?? {};
      let changed = false;
      for (const dim of resetDimensions) {
        if (scores[dim]) {
          delete scores[dim];
          changed = true;
        }
      }
      if (changed) {
        await prisma.endpointTaskPerformance.update({
          where: { id: p.id },
          data: { dimensionScores: scores },
        });
      }
    }
  }
}
```

- [ ] **Step 4: Run tests to verify they pass**

Run: `cd apps/web && npx vitest run lib/routing/production-feedback.test.ts`

- [ ] **Step 5: Update barrel export**

Add to `apps/web/lib/routing/index.ts`:

```typescript
export { updateEndpointDimensionScores } from "./production-feedback";
```

- [ ] **Step 6: Commit**

```bash
git add apps/web/lib/routing/
git commit -m "feat: implement production observation feedback with two-stage propagation (EP-INF-001-P6)"
```

---

## Task 5: Wire Into Orchestrator-Evaluator

**Files:**
- Modify: `apps/web/lib/orchestrator-evaluator.ts`

- [ ] **Step 1: Read current orchestrator-evaluator.ts**

Read the file to find where `updatePerformanceProfile` is called after scoring.

- [ ] **Step 2: Add import and call**

Add import at top of `apps/web/lib/orchestrator-evaluator.ts`:

```typescript
import { updateEndpointDimensionScores } from "./routing/production-feedback";
```

Find the call to `updatePerformanceProfile(...)` (inside `evaluateAndUpdateProfile`). Immediately after it, add:

```typescript
    // EP-INF-001-P6: Feed orchestrator score into routing dimension profiles
    await updateEndpointDimensionScores(input.endpointId, input.taskType, score).catch((err) =>
      console.error("[orchestrator-evaluator] dimension score update failed:", err),
    );
```

This is fire-and-forget with error swallowing, matching the existing pattern.

- [ ] **Step 3: Commit**

```bash
git add apps/web/lib/orchestrator-evaluator.ts
git commit -m "feat: wire production observation feedback into orchestrator-evaluator (EP-INF-001-P6)"
```

---

## Task 6: Server Action for Ops UI Trigger

**Files:**
- Modify: `apps/web/lib/actions/endpoint-performance.ts`

- [ ] **Step 1: Read current endpoint-performance.ts**

Read the file to understand the existing server action pattern.

- [ ] **Step 2: Add triggerDimensionEval action**

Add after the existing `triggerEndpointTests` function:

```typescript
export async function triggerDimensionEval(endpointId?: string) {
  const userId = await requireViewAccess();
  const session = await auth();
  if (!session?.user || !can({ platformRole: session.user.platformRole, isSuperuser: session.user.isSuperuser }, "manage_capabilities")) {
    throw new Error("Running dimension eval requires manage_capabilities permission");
  }

  if (endpointId) {
    const { runDimensionEval } = await import("@/lib/routing/eval-runner");
    return runDimensionEval(endpointId, userId);
  } else {
    const { runAllDimensionEvals } = await import("@/lib/routing/eval-runner");
    return runAllDimensionEvals(userId);
  }
}
```

- [ ] **Step 3: Commit**

```bash
git add apps/web/lib/actions/endpoint-performance.ts
git commit -m "feat: add triggerDimensionEval server action for ops UI (EP-INF-001-P6)"
```

---

## Task 7: Run Full Test Suite & Verify

- [ ] **Step 1: Run all routing tests**

Run: `cd apps/web && npx vitest run lib/routing/`

Expected: All tests pass — scoring (8), pipeline (13), eval-scoring (new), eval-runner (new), production-feedback (new).

- [ ] **Step 2: Check TypeScript compilation**

Run: `cd apps/web && npx tsc --noEmit 2>&1 | grep routing/`

Expected: Zero errors in routing module.

- [ ] **Step 3: Commit if any fixups needed**

---

## Summary

After completing these 7 tasks:

1. **Golden test registry** — ~25 deterministic prompts across 7 dimensions in `golden-tests.ts`
2. **Eval scoring** — 7 scoring functions (exact, partial, schema, tool_call, structural, retrieval, orchestrator) with TDD
3. **Eval runner** — orchestrates golden tests, computes dimension scores with rolling average, detects drift (15pt warning, 25pt severe), updates `ModelProvider` profiles, persists `EndpointTestRun` audit records
4. **Production feedback** — maps orchestrator scores to dimension nudges, two-stage accumulation/propagation, guardrails (24hr protection, 5-observation minimum, clamping)
5. **Orchestrator integration** — existing scoring pipeline now feeds dimension profiles
6. **Server action** — `triggerDimensionEval` for ops UI (per-endpoint or all)
