# Fully Parallel Evaluation Results

Generated: 2026-01-11T06:58:49.010Z
Total Time: 346.1s
Suite: pack
Models: 8
Scenarios: 24 of 24 (limit=0)
Total evaluations: 192

## Summary by Model

| Model | Total | Passed | Failed | Avg Time (s) |
|-------|-------|--------|--------|--------------|
| claude-haiku-4.5 | 24 | 24 | 0 | 3.6 |
| gpt-5-mini | 24 | 24 | 0 | 83.0 |
| gemini-3-flash | 24 | 24 | 0 | 15.2 |
| deepseek-r1 | 24 | 24 | 0 | 70.2 |
| deepseek-v3.2 | 24 | 24 | 0 | 99.2 |
| qwen3-235b | 24 | 24 | 0 | 116.0 |
| minimax-m2.1 | 24 | 24 | 0 | 57.3 |
| mimo-v2-flash-free | 24 | 24 | 0 | 48.6 |

## Summary by Scenario

| Scenario | Total | Passed | Failed |
|----------|-------|--------|--------|
| Next: academic tool-driven literature debrief | 8 | 8 | 0 |
| Next: academic vague anchor | 8 | 8 | 0 |
| Next: banker tool-driven outbound pack | 8 | 8 | 0 |
| Next: banker vague (fast debrief) | 8 | 8 | 0 |
| Next: CTO tool-driven CVE plan | 8 | 8 | 0 |
| Next: CTO vague exposure | 8 | 8 | 0 |
| Next: ecosystem tool-driven second-order brief | 8 | 8 | 0 |
| Next: ecosystem vague incident | 8 | 8 | 0 |
| Next: exec tool-driven cost model | 8 | 8 | 0 |
| Next: exec vague standardize | 8 | 8 | 0 |
| Next: founder tool-driven memo | 8 | 8 | 0 |
| Next: founder vague positioning | 8 | 8 | 0 |
| Next: product tool-driven expandable card schema | 8 | 8 | 0 |
| Next: product vague UI usable | 8 | 8 | 0 |
| Next: quant tool-driven signal set JSON | 8 | 8 | 0 |
| Next: quant vague what to track | 8 | 8 | 0 |
| Next: sales tool-driven one-screen + objections | 8 | 8 | 0 |
| Next: sales vague shareable | 8 | 8 | 0 |
| Next: VC tool-driven comps + diligence | 8 | 8 | 0 |
| Next: VC vague wedge | 8 | 8 | 0 |
| Pack: exec cross-provider pricing comparison | 8 | 8 | 0 |
| Pack: meta budgeted deep dive (<=3 tools, <=$0.25) | 8 | 8 | 0 |
| Stress: ambiguous persona (wedge + outreach) | 8 | 8 | 0 |
| Stress: contradiction handling (Seed vs Series A) | 8 | 8 | 0 |

## Detailed Results

### claude-haiku-4.5

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| Next: banker vague (fast debrief) | ✅ PASS | 1.4 | - |
| Next: banker tool-driven outbound pack | ✅ PASS | 1.8 | - |
| Next: VC vague wedge | ✅ PASS | 2.1 | - |
| Next: VC tool-driven comps + diligence | ✅ PASS | 3.9 | - |
| Next: CTO vague exposure | ✅ PASS | 1.4 | - |
| Next: CTO tool-driven CVE plan | ✅ PASS | 3.9 | - |
| Next: founder vague positioning | ✅ PASS | 4.0 | - |
| Next: founder tool-driven memo | ✅ PASS | 4.3 | - |
| Next: academic vague anchor | ✅ PASS | 3.9 | - |
| Next: academic tool-driven literature debrief | ✅ PASS | 4.1 | - |
| Next: exec vague standardize | ✅ PASS | 4.1 | - |
| Next: exec tool-driven cost model | ✅ PASS | 3.7 | - |
| Next: ecosystem vague incident | ✅ PASS | 4.1 | - |
| Next: ecosystem tool-driven second-order brief | ✅ PASS | 3.9 | - |
| Next: quant vague what to track | ✅ PASS | 4.2 | - |
| Next: quant tool-driven signal set JSON | ✅ PASS | 3.9 | - |
| Next: product vague UI usable | ✅ PASS | 4.0 | - |
| Next: product tool-driven expandable card schema | ✅ PASS | 4.1 | - |
| Next: sales vague shareable | ✅ PASS | 4.1 | - |
| Next: sales tool-driven one-screen + objections | ✅ PASS | 4.1 | - |
| Stress: ambiguous persona (wedge + outreach) | ✅ PASS | 4.0 | - |
| Stress: contradiction handling (Seed vs Series A) | ✅ PASS | 3.7 | - |
| Pack: exec cross-provider pricing comparison | ✅ PASS | 4.0 | - |
| Pack: meta budgeted deep dive (<=3 tools, <=$0.25) | ✅ PASS | 4.1 | - |

### gpt-5-mini

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| Next: banker vague (fast debrief) | ✅ PASS | 55.2 | - |
| Next: banker tool-driven outbound pack | ✅ PASS | 78.4 | - |
| Next: VC vague wedge | ✅ PASS | 54.1 | - |
| Next: VC tool-driven comps + diligence | ✅ PASS | 89.8 | - |
| Next: CTO vague exposure | ✅ PASS | 77.9 | - |
| Next: CTO tool-driven CVE plan | ✅ PASS | 101.6 | - |
| Next: founder vague positioning | ✅ PASS | 83.0 | - |
| Next: founder tool-driven memo | ✅ PASS | 73.5 | - |
| Next: academic vague anchor | ✅ PASS | 61.1 | - |
| Next: academic tool-driven literature debrief | ✅ PASS | 93.8 | - |
| Next: exec vague standardize | ✅ PASS | 49.0 | - |
| Next: exec tool-driven cost model | ✅ PASS | 98.1 | - |
| Next: ecosystem vague incident | ✅ PASS | 67.8 | - |
| Next: ecosystem tool-driven second-order brief | ✅ PASS | 96.0 | - |
| Next: quant vague what to track | ✅ PASS | 346.1 | - |
| Next: quant tool-driven signal set JSON | ✅ PASS | 60.0 | - |
| Next: product vague UI usable | ✅ PASS | 60.0 | - |
| Next: product tool-driven expandable card schema | ✅ PASS | 93.0 | - |
| Next: sales vague shareable | ✅ PASS | 46.9 | - |
| Next: sales tool-driven one-screen + objections | ✅ PASS | 70.4 | - |
| Stress: ambiguous persona (wedge + outreach) | ✅ PASS | 59.9 | - |
| Stress: contradiction handling (Seed vs Series A) | ✅ PASS | 56.2 | - |
| Pack: exec cross-provider pricing comparison | ✅ PASS | 59.4 | - |
| Pack: meta budgeted deep dive (<=3 tools, <=$0.25) | ✅ PASS | 61.9 | - |

### gemini-3-flash

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| Next: banker vague (fast debrief) | ✅ PASS | 15.1 | - |
| Next: banker tool-driven outbound pack | ✅ PASS | 16.7 | - |
| Next: VC vague wedge | ✅ PASS | 15.1 | - |
| Next: VC tool-driven comps + diligence | ✅ PASS | 24.2 | - |
| Next: CTO vague exposure | ✅ PASS | 14.1 | - |
| Next: CTO tool-driven CVE plan | ✅ PASS | 14.5 | - |
| Next: founder vague positioning | ✅ PASS | 15.2 | - |
| Next: founder tool-driven memo | ✅ PASS | 14.0 | - |
| Next: academic vague anchor | ✅ PASS | 15.0 | - |
| Next: academic tool-driven literature debrief | ✅ PASS | 15.2 | - |
| Next: exec vague standardize | ✅ PASS | 14.3 | - |
| Next: exec tool-driven cost model | ✅ PASS | 15.2 | - |
| Next: ecosystem vague incident | ✅ PASS | 15.4 | - |
| Next: ecosystem tool-driven second-order brief | ✅ PASS | 14.1 | - |
| Next: quant vague what to track | ✅ PASS | 14.7 | - |
| Next: quant tool-driven signal set JSON | ✅ PASS | 14.3 | - |
| Next: product vague UI usable | ✅ PASS | 14.0 | - |
| Next: product tool-driven expandable card schema | ✅ PASS | 14.7 | - |
| Next: sales vague shareable | ✅ PASS | 14.4 | - |
| Next: sales tool-driven one-screen + objections | ✅ PASS | 15.7 | - |
| Stress: ambiguous persona (wedge + outreach) | ✅ PASS | 14.2 | - |
| Stress: contradiction handling (Seed vs Series A) | ✅ PASS | 15.2 | - |
| Pack: exec cross-provider pricing comparison | ✅ PASS | 14.8 | - |
| Pack: meta budgeted deep dive (<=3 tools, <=$0.25) | ✅ PASS | 15.9 | - |

### deepseek-r1

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| Next: banker vague (fast debrief) | ✅ PASS | 42.0 | - |
| Next: banker tool-driven outbound pack | ✅ PASS | 42.3 | - |
| Next: VC vague wedge | ✅ PASS | 95.2 | - |
| Next: VC tool-driven comps + diligence | ✅ PASS | 250.6 | - |
| Next: CTO vague exposure | ✅ PASS | 60.3 | - |
| Next: CTO tool-driven CVE plan | ✅ PASS | 139.2 | - |
| Next: founder vague positioning | ✅ PASS | 27.3 | - |
| Next: founder tool-driven memo | ✅ PASS | 198.9 | - |
| Next: academic vague anchor | ✅ PASS | 10.7 | - |
| Next: academic tool-driven literature debrief | ✅ PASS | 99.9 | - |
| Next: exec vague standardize | ✅ PASS | 68.4 | - |
| Next: exec tool-driven cost model | ✅ PASS | 75.9 | - |
| Next: ecosystem vague incident | ✅ PASS | 12.4 | - |
| Next: ecosystem tool-driven second-order brief | ✅ PASS | 130.4 | - |
| Next: quant vague what to track | ✅ PASS | 72.5 | - |
| Next: quant tool-driven signal set JSON | ✅ PASS | 17.2 | - |
| Next: product vague UI usable | ✅ PASS | 17.1 | - |
| Next: product tool-driven expandable card schema | ✅ PASS | 18.9 | - |
| Next: sales vague shareable | ✅ PASS | 47.6 | - |
| Next: sales tool-driven one-screen + objections | ✅ PASS | 13.3 | - |
| Stress: ambiguous persona (wedge + outreach) | ✅ PASS | 52.1 | - |
| Stress: contradiction handling (Seed vs Series A) | ✅ PASS | 12.0 | - |
| Pack: exec cross-provider pricing comparison | ✅ PASS | 111.7 | - |
| Pack: meta budgeted deep dive (<=3 tools, <=$0.25) | ✅ PASS | 69.1 | - |

### deepseek-v3.2

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| Next: banker vague (fast debrief) | ✅ PASS | 145.0 | - |
| Next: banker tool-driven outbound pack | ✅ PASS | 108.8 | - |
| Next: VC vague wedge | ✅ PASS | 114.9 | - |
| Next: VC tool-driven comps + diligence | ✅ PASS | 69.1 | - |
| Next: CTO vague exposure | ✅ PASS | 83.4 | - |
| Next: CTO tool-driven CVE plan | ✅ PASS | 102.6 | - |
| Next: founder vague positioning | ✅ PASS | 65.8 | - |
| Next: founder tool-driven memo | ✅ PASS | 13.6 | - |
| Next: academic vague anchor | ✅ PASS | 66.8 | - |
| Next: academic tool-driven literature debrief | ✅ PASS | 119.0 | - |
| Next: exec vague standardize | ✅ PASS | 145.2 | - |
| Next: exec tool-driven cost model | ✅ PASS | 238.5 | - |
| Next: ecosystem vague incident | ✅ PASS | 14.9 | - |
| Next: ecosystem tool-driven second-order brief | ✅ PASS | 54.8 | - |
| Next: quant vague what to track | ✅ PASS | 130.0 | - |
| Next: quant tool-driven signal set JSON | ✅ PASS | 23.2 | - |
| Next: product vague UI usable | ✅ PASS | 77.0 | - |
| Next: product tool-driven expandable card schema | ✅ PASS | 30.7 | - |
| Next: sales vague shareable | ✅ PASS | 120.4 | - |
| Next: sales tool-driven one-screen + objections | ✅ PASS | 121.9 | - |
| Stress: ambiguous persona (wedge + outreach) | ✅ PASS | 125.9 | - |
| Stress: contradiction handling (Seed vs Series A) | ✅ PASS | 79.0 | - |
| Pack: exec cross-provider pricing comparison | ✅ PASS | 227.2 | - |
| Pack: meta budgeted deep dive (<=3 tools, <=$0.25) | ✅ PASS | 102.5 | - |

### qwen3-235b

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| Next: banker vague (fast debrief) | ✅ PASS | 121.8 | - |
| Next: banker tool-driven outbound pack | ✅ PASS | 145.0 | - |
| Next: VC vague wedge | ✅ PASS | 132.7 | - |
| Next: VC tool-driven comps + diligence | ✅ PASS | 107.0 | - |
| Next: CTO vague exposure | ✅ PASS | 101.6 | - |
| Next: CTO tool-driven CVE plan | ✅ PASS | 142.1 | - |
| Next: founder vague positioning | ✅ PASS | 127.9 | - |
| Next: founder tool-driven memo | ✅ PASS | 119.6 | - |
| Next: academic vague anchor | ✅ PASS | 142.7 | - |
| Next: academic tool-driven literature debrief | ✅ PASS | 57.4 | - |
| Next: exec vague standardize | ✅ PASS | 121.4 | - |
| Next: exec tool-driven cost model | ✅ PASS | 95.8 | - |
| Next: ecosystem vague incident | ✅ PASS | 117.2 | - |
| Next: ecosystem tool-driven second-order brief | ✅ PASS | 125.4 | - |
| Next: quant vague what to track | ✅ PASS | 36.4 | - |
| Next: quant tool-driven signal set JSON | ✅ PASS | 181.6 | - |
| Next: product vague UI usable | ✅ PASS | 163.6 | - |
| Next: product tool-driven expandable card schema | ✅ PASS | 167.4 | - |
| Next: sales vague shareable | ✅ PASS | 133.5 | - |
| Next: sales tool-driven one-screen + objections | ✅ PASS | 144.0 | - |
| Stress: ambiguous persona (wedge + outreach) | ✅ PASS | 36.2 | - |
| Stress: contradiction handling (Seed vs Series A) | ✅ PASS | 109.5 | - |
| Pack: exec cross-provider pricing comparison | ✅ PASS | 125.5 | - |
| Pack: meta budgeted deep dive (<=3 tools, <=$0.25) | ✅ PASS | 28.5 | - |

### minimax-m2.1

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| Next: banker vague (fast debrief) | ✅ PASS | 30.8 | - |
| Next: banker tool-driven outbound pack | ✅ PASS | 54.7 | - |
| Next: VC vague wedge | ✅ PASS | 44.7 | - |
| Next: VC tool-driven comps + diligence | ✅ PASS | 122.5 | - |
| Next: CTO vague exposure | ✅ PASS | 42.8 | - |
| Next: CTO tool-driven CVE plan | ✅ PASS | 90.0 | - |
| Next: founder vague positioning | ✅ PASS | 81.6 | - |
| Next: founder tool-driven memo | ✅ PASS | 145.0 | - |
| Next: academic vague anchor | ✅ PASS | 66.4 | - |
| Next: academic tool-driven literature debrief | ✅ PASS | 74.3 | - |
| Next: exec vague standardize | ✅ PASS | 61.4 | - |
| Next: exec tool-driven cost model | ✅ PASS | 51.0 | - |
| Next: ecosystem vague incident | ✅ PASS | 25.6 | - |
| Next: ecosystem tool-driven second-order brief | ✅ PASS | 45.9 | - |
| Next: quant vague what to track | ✅ PASS | 29.4 | - |
| Next: quant tool-driven signal set JSON | ✅ PASS | 32.8 | - |
| Next: product vague UI usable | ✅ PASS | 20.7 | - |
| Next: product tool-driven expandable card schema | ✅ PASS | 57.0 | - |
| Next: sales vague shareable | ✅ PASS | 29.0 | - |
| Next: sales tool-driven one-screen + objections | ✅ PASS | 39.5 | - |
| Stress: ambiguous persona (wedge + outreach) | ✅ PASS | 30.0 | - |
| Stress: contradiction handling (Seed vs Series A) | ✅ PASS | 54.5 | - |
| Pack: exec cross-provider pricing comparison | ✅ PASS | 114.5 | - |
| Pack: meta budgeted deep dive (<=3 tools, <=$0.25) | ✅ PASS | 31.3 | - |

### mimo-v2-flash-free

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| Next: banker vague (fast debrief) | ✅ PASS | 20.8 | - |
| Next: banker tool-driven outbound pack | ✅ PASS | 37.5 | - |
| Next: VC vague wedge | ✅ PASS | 29.4 | - |
| Next: VC tool-driven comps + diligence | ✅ PASS | 162.2 | - |
| Next: CTO vague exposure | ✅ PASS | 23.6 | - |
| Next: CTO tool-driven CVE plan | ✅ PASS | 54.8 | - |
| Next: founder vague positioning | ✅ PASS | 39.5 | - |
| Next: founder tool-driven memo | ✅ PASS | 88.8 | - |
| Next: academic vague anchor | ✅ PASS | 27.0 | - |
| Next: academic tool-driven literature debrief | ✅ PASS | 81.1 | - |
| Next: exec vague standardize | ✅ PASS | 23.9 | - |
| Next: exec tool-driven cost model | ✅ PASS | 62.3 | - |
| Next: ecosystem vague incident | ✅ PASS | 29.0 | - |
| Next: ecosystem tool-driven second-order brief | ✅ PASS | 69.8 | - |
| Next: quant vague what to track | ✅ PASS | 31.1 | - |
| Next: quant tool-driven signal set JSON | ✅ PASS | 107.5 | - |
| Next: product vague UI usable | ✅ PASS | 21.5 | - |
| Next: product tool-driven expandable card schema | ✅ PASS | 35.0 | - |
| Next: sales vague shareable | ✅ PASS | 25.3 | - |
| Next: sales tool-driven one-screen + objections | ✅ PASS | 40.3 | - |
| Stress: ambiguous persona (wedge + outreach) | ✅ PASS | 28.3 | - |
| Stress: contradiction handling (Seed vs Series A) | ✅ PASS | 22.7 | - |
| Pack: exec cross-provider pricing comparison | ✅ PASS | 55.0 | - |
| Pack: meta budgeted deep dive (<=3 tools, <=$0.25) | ✅ PASS | 49.4 | - |

## Disclosure Metrics (P0 Instrumentation)

### Disclosure Level by Model

| Model | Full | Partial | None |
|-------|------|---------|------|
| claude-haiku-4.5 | 0 | 24 | 0 |
| gpt-5-mini | 0 | 24 | 0 |
| gemini-3-flash | 0 | 24 | 0 |
| deepseek-r1 | 1 | 23 | 0 |
| deepseek-v3.2 | 0 | 24 | 0 |
| qwen3-235b | 0 | 24 | 0 |
| minimax-m2.1 | 0 | 24 | 0 |
| mimo-v2-flash-free | 0 | 24 | 0 |

### Disclosure Warnings (Non-Scored)

> **Note:** These warnings are informational and do not affect pass/fail.
> They track progressive disclosure usage for Week 1-2 observability.

⚠️ [deepseek-r1/pack_exec_cross_provider_pricing] Excessive direct tool calls: 109 (>10)
⚠️ [qwen3-235b/next_ecosystem_vague_soundcloud_vpn] Excessive direct tool calls: 11 (>10)
⚠️ [qwen3-235b/next_ecosystem_tool_second_order_brief] Excessive direct tool calls: 12 (>10)
