# Fully Parallel Evaluation Results

Generated: 2026-01-10T22:35:36.304Z
Total Time: 301.2s
Suite: pack
Models: 8
Scenarios: 100 of 100 (limit=0)
Total evaluations: 800

## Summary by Model

| Model | Total | Passed | Failed | Avg Time (s) |
|-------|-------|--------|--------|--------------|
| claude-haiku-4.5 | 100 | 0 | 99 | 66.5 |
| gpt-5-mini | 100 | 0 | 99 | 66.9 |
| gemini-3-flash | 100 | 0 | 100 | 30.3 |
| deepseek-r1 | 100 | 0 | 96 | 37.4 |
| deepseek-v3.2 | 100 | 0 | 100 | 10.7 |
| qwen3-235b | 100 | 0 | 100 | 10.6 |
| minimax-m2.1 | 100 | 0 | 100 | 10.6 |
| mimo-v2-flash-free | 100 | 0 | 100 | 11.4 |

## Summary by Scenario

| Scenario | Total | Passed | Failed |
|----------|-------|--------|--------|
| Academic: Alzheimer's Mechanism | 3 | 0 | 3 |
| Academic: Animal Model Validation | 3 | 0 | 3 |
| Academic: Antibiotic Resistance | 3 | 0 | 3 |
| Academic: Biomarker Discovery | 3 | 0 | 3 |
| Academic: Cancer Immunotherapy | 3 | 0 | 3 |
| Academic: CAR-T Therapy Paper | 2 | 0 | 2 |
| Academic: Cardiomyopathy Treatment | 3 | 0 | 3 |
| Academic: Clinical Trial Design | 3 | 0 | 3 |
| Academic: CRISPR Methodology | 3 | 0 | 3 |
| Academic: CRISPR Paper Summary | 3 | 0 | 3 |
| Academic: Flow Cytometry Standards | 3 | 0 | 3 |
| Academic: Gene Therapy Advances | 3 | 0 | 3 |
| Academic: Genomic Analysis | 3 | 0 | 3 |
| Academic: Long COVID Research | 3 | 0 | 3 |
| Academic: Medical Imaging Analysis | 3 | 0 | 3 |
| Academic: Meta-Analysis Quality | 3 | 0 | 3 |
| Academic: Microbiome Research | 3 | 0 | 3 |
| Academic: mRNA Vaccine Paper | 3 | 0 | 3 |
| Academic: Neurodegeneration Research | 3 | 0 | 3 |
| Academic: PD-L1 Research | 3 | 0 | 3 |
| Academic: Protein Folding | 3 | 0 | 3 |
| Academic: Replication Study | 3 | 0 | 3 |
| Academic: RNA-Seq Validation | 3 | 0 | 3 |
| Academic: Senolytics Research | 3 | 0 | 3 |
| Academic: Stem Cell Therapy | 3 | 0 | 3 |
| Financial: AI Chip Market | 3 | 0 | 3 |
| Financial: Alphabet 2024 Revenue | 4 | 0 | 4 |
| Financial: Amazon 2024 Revenue | 3 | 0 | 3 |
| Financial: Amazon 2024 Risk Factors | 4 | 0 | 4 |
| Financial: AMD 2024 Revenue | 4 | 0 | 4 |
| Financial: AMD 2024 Risk Factors | 4 | 0 | 4 |
| Financial: AMD Business Description | 4 | 0 | 4 |
| Financial: Apple 2024 Revenue | 4 | 0 | 4 |
| Financial: Apple 2024 Risk Factors | 4 | 0 | 4 |
| Financial: Cloud Business Models | 3 | 0 | 3 |
| Financial: Cloud Growth Rates | 3 | 0 | 3 |
| Financial: Cloud Market Comparison | 3 | 0 | 3 |
| Financial: Google 2024 Risk Factors | 3 | 0 | 3 |
| Financial: Intel 2024 Revenue | 4 | 0 | 4 |
| Financial: Intel 2024 Risk Factors | 3 | 0 | 3 |
| Financial: Meta 2024 Revenue | 4 | 0 | 4 |
| Financial: Meta 2024 Risk Factors | 3 | 0 | 3 |
| Financial: Microsoft 2024 Revenue | 4 | 0 | 4 |
| Financial: Microsoft 2024 Risk Factors | 4 | 0 | 4 |
| Financial: NVIDIA 2024 Revenue | 4 | 0 | 4 |
| Financial: NVIDIA 2024 Risk Factors | 3 | 0 | 3 |
| Financial: NVIDIA Business Description | 3 | 0 | 3 |
| Financial: Qualcomm 2024 Revenue | 4 | 0 | 4 |
| Financial: Qualcomm 2024 Risk Factors | 3 | 0 | 3 |
| Financial: Semiconductor Business Models | 4 | 0 | 4 |
| Financial: Semiconductor Market Share | 3 | 0 | 3 |
| Financial: Social Media Business Models | 4 | 0 | 4 |
| Financial: Social Media Comparison | 3 | 0 | 3 |
| Financial: Tesla 2024 Revenue | 4 | 0 | 4 |
| Financial: Tesla 2024 Risk Factors | 4 | 0 | 4 |
| Market: Angular vs Vue Comparison | 3 | 0 | 3 |
| Market: Container Orchestration Comparison | 3 | 0 | 3 |
| Market: Docker vs Podman Comparison | 3 | 0 | 3 |
| Market: Electron vs Tauri Comparison | 3 | 0 | 3 |
| Market: GraphQL vs REST API Frameworks | 2 | 0 | 2 |
| Market: MongoDB vs PostgreSQL Comparison | 3 | 0 | 3 |
| Market: Next.js vs Nuxt Comparison | 3 | 0 | 3 |
| Market: PostgreSQL vs MySQL Comparison | 3 | 0 | 3 |
| Market: React vs Vue Comparison | 3 | 0 | 3 |
| Market: TensorFlow vs PyTorch | 2 | 0 | 2 |
| Market: Trending AI JavaScript Repos | 3 | 0 | 3 |
| Market: Trending Blockchain Repos | 3 | 0 | 3 |
| Market: Trending Database Repos | 3 | 0 | 3 |
| Market: Trending DevOps Tools | 3 | 0 | 3 |
| Market: Trending Go Cloud Repos | 3 | 0 | 3 |
| Market: Trending Mobile Dev Repos | 3 | 0 | 3 |
| Market: Trending Python AI Repos | 3 | 0 | 3 |
| Market: Trending Runtime Repos | 3 | 0 | 3 |
| Market: Trending Rust Systems Repos | 3 | 0 | 3 |
| Market: Trending TypeScript Frontend | 3 | 0 | 3 |
| offset:0 | 4 | 0 | 4 |
| offset:1 | 4 | 0 | 4 |
| offset:10 | 5 | 0 | 5 |
| offset:11 | 5 | 0 | 5 |
| offset:12 | 4 | 0 | 4 |
| offset:13 | 4 | 0 | 4 |
| offset:14 | 4 | 0 | 4 |
| offset:15 | 4 | 0 | 4 |
| offset:16 | 5 | 0 | 5 |
| offset:17 | 4 | 0 | 4 |
| offset:18 | 5 | 0 | 5 |
| offset:19 | 5 | 0 | 5 |
| offset:2 | 4 | 0 | 4 |
| offset:20 | 4 | 0 | 4 |
| offset:21 | 5 | 0 | 5 |
| offset:22 | 5 | 0 | 5 |
| offset:23 | 4 | 0 | 4 |
| offset:24 | 4 | 0 | 4 |
| offset:25 | 5 | 0 | 5 |
| offset:26 | 5 | 0 | 5 |
| offset:27 | 5 | 0 | 5 |
| offset:28 | 5 | 0 | 5 |
| offset:29 | 5 | 0 | 5 |
| offset:3 | 4 | 0 | 4 |
| offset:30 | 5 | 0 | 5 |
| offset:31 | 5 | 0 | 5 |
| offset:32 | 5 | 0 | 5 |
| offset:33 | 5 | 0 | 5 |
| offset:34 | 5 | 0 | 5 |
| offset:35 | 5 | 0 | 5 |
| offset:36 | 5 | 0 | 5 |
| offset:37 | 5 | 0 | 5 |
| offset:38 | 5 | 0 | 5 |
| offset:39 | 5 | 0 | 5 |
| offset:4 | 4 | 0 | 4 |
| offset:40 | 5 | 0 | 5 |
| offset:41 | 5 | 0 | 5 |
| offset:42 | 5 | 0 | 5 |
| offset:43 | 5 | 0 | 5 |
| offset:44 | 5 | 0 | 5 |
| offset:45 | 5 | 0 | 5 |
| offset:46 | 5 | 0 | 5 |
| offset:47 | 5 | 0 | 5 |
| offset:48 | 5 | 0 | 5 |
| offset:49 | 5 | 0 | 5 |
| offset:5 | 5 | 0 | 5 |
| offset:50 | 4 | 0 | 4 |
| offset:51 | 5 | 0 | 5 |
| offset:52 | 5 | 0 | 5 |
| offset:53 | 5 | 0 | 5 |
| offset:54 | 6 | 0 | 6 |
| offset:55 | 5 | 0 | 5 |
| offset:56 | 5 | 0 | 5 |
| offset:57 | 6 | 0 | 6 |
| offset:58 | 5 | 0 | 5 |
| offset:59 | 5 | 0 | 5 |
| offset:6 | 4 | 0 | 4 |
| offset:60 | 5 | 0 | 5 |
| offset:61 | 5 | 0 | 5 |
| offset:62 | 5 | 0 | 5 |
| offset:63 | 5 | 0 | 5 |
| offset:64 | 5 | 0 | 5 |
| offset:65 | 5 | 0 | 5 |
| offset:66 | 5 | 0 | 5 |
| offset:67 | 5 | 0 | 5 |
| offset:68 | 5 | 0 | 5 |
| offset:69 | 5 | 0 | 5 |
| offset:7 | 4 | 0 | 4 |
| offset:70 | 5 | 0 | 5 |
| offset:71 | 5 | 0 | 5 |
| offset:72 | 5 | 0 | 5 |
| offset:73 | 5 | 0 | 5 |
| offset:74 | 5 | 0 | 5 |
| offset:75 | 5 | 0 | 5 |
| offset:76 | 5 | 0 | 5 |
| offset:77 | 5 | 0 | 5 |
| offset:78 | 5 | 0 | 5 |
| offset:79 | 5 | 0 | 5 |
| offset:8 | 4 | 0 | 4 |
| offset:80 | 5 | 0 | 5 |
| offset:81 | 5 | 0 | 5 |
| offset:82 | 5 | 0 | 5 |
| offset:83 | 5 | 0 | 5 |
| offset:84 | 5 | 0 | 5 |
| offset:85 | 5 | 0 | 5 |
| offset:86 | 5 | 0 | 5 |
| offset:87 | 5 | 0 | 5 |
| offset:88 | 5 | 0 | 5 |
| offset:89 | 5 | 0 | 5 |
| offset:9 | 4 | 0 | 4 |
| offset:90 | 5 | 0 | 5 |
| offset:91 | 6 | 0 | 6 |
| offset:92 | 5 | 0 | 5 |
| offset:93 | 5 | 0 | 5 |
| offset:94 | 5 | 0 | 5 |
| offset:95 | 5 | 0 | 5 |
| offset:96 | 5 | 0 | 5 |
| offset:97 | 5 | 0 | 5 |
| offset:98 | 6 | 0 | 6 |
| offset:99 | 5 | 0 | 5 |
| Security: Auth Bypass Remediation | 3 | 0 | 3 |
| Security: Authentication Bypass | 3 | 0 | 3 |
| Security: Cloud Vulnerability Impact | 3 | 0 | 3 |
| Security: Critical RCE Remediation | 3 | 0 | 3 |
| Security: Critical Vulnerability Scoring | 3 | 0 | 3 |
| Security: CVE-2023-36025 Remediation | 3 | 0 | 3 |
| Security: CVE-2024-21887 Remediation | 3 | 0 | 3 |
| Security: CVE-2024-21887 Scoring | 3 | 0 | 3 |
| Security: CVE-2024-23897 Scoring | 3 | 0 | 3 |
| Security: Data Breach Analysis | 3 | 0 | 3 |
| Security: FortiOS CVE | 3 | 0 | 3 |
| Security: FortiOS Impact Analysis | 4 | 0 | 4 |
| Security: FortiOS Patching | 3 | 0 | 3 |
| Security: HTTP/2 Rapid Reset | 3 | 0 | 3 |
| Security: HTTP/2 Rapid Reset Mitigation | 3 | 0 | 3 |
| Security: Jenkins CVE Remediation | 3 | 0 | 3 |
| Security: Path Traversal CVE | 3 | 0 | 3 |
| Security: Path Traversal Mitigation | 3 | 0 | 3 |
| Security: Ransomware Impact Assessment | 3 | 0 | 3 |
| Security: RCE Vulnerability | 3 | 0 | 3 |
| Security: SQL Injection CVE | 3 | 0 | 3 |
| Security: SQL Injection Response | 3 | 0 | 3 |
| Security: Supply Chain Response | 3 | 0 | 3 |
| Security: Third-Party Risk | 2 | 0 | 2 |
| Security: Windows SmartScreen CVE | 3 | 0 | 3 |

## Detailed Results

### claude-haiku-4.5

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| Financial: Tesla 2024 Revenue | ❌ FAIL | 118.5 | persona mismatch: got QUANT_ANALYST expected JPM_STARTUP_BANKER |
| Financial: Apple 2024 Revenue | ❌ FAIL | 68.1 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Microsoft 2024 Revenue | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Alphabet 2024 Revenue | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: NVIDIA 2024 Revenue | ❌ FAIL | 68.2 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Amazon 2024 Revenue | ❌ FAIL | 162.7 | persona mismatch: got QUANT_ANALYST expected JPM_STARTUP_BANKER |
| Financial: Meta 2024 Revenue | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: AMD 2024 Revenue | ❌ FAIL | 109.6 | unknown ground truth entity: 0000002488 |
| Financial: Intel 2024 Revenue | ❌ FAIL | 61.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Qualcomm 2024 Revenue | ❌ FAIL | 141.2 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Google 2024 Risk Factors | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Financial: NVIDIA 2024 Risk Factors | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Apple 2024 Risk Factors | ❌ FAIL | 61.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Microsoft 2024 Risk Factors | ❌ FAIL | 106.4 | persona mismatch: got ENTERPRISE_EXEC expected JPM_STARTUP_BANKER |
| Financial: Tesla 2024 Risk Factors | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Amazon 2024 Risk Factors | ❌ FAIL | 5.0 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Meta 2024 Risk Factors | ❌ FAIL | 124.0 | persona mismatch: got ENTERPRISE_EXEC expected FOUNDER_STRATEGY |
| Financial: AMD 2024 Risk Factors | ❌ FAIL | 131.5 | persona mismatch: got ENTERPRISE_EXEC expected JPM_STARTUP_BANKER |
| Financial: Intel 2024 Risk Factors | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Qualcomm 2024 Risk Factors | ❌ FAIL | 62.6 | Missing [DEBRIEF_V1_JSON] block |
| Financial: AMD Business Description | ❌ FAIL | 166.1 | Missing [DEBRIEF_V1_JSON] block |
| Financial: NVIDIA Business Description | ❌ FAIL | 63.0 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Cloud Business Models | ❌ FAIL | 61.9 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Semiconductor Business Models | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Social Media Business Models | ❌ FAIL | 62.4 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Cloud Market Comparison | ❌ FAIL | 95.1 | Missing [DEBRIEF_V1_JSON] block |
| Financial: AI Chip Market | ❌ FAIL | 5.0 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Social Media Comparison | ❌ FAIL | 67.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Cloud Growth Rates | ❌ FAIL | 101.7 | persona mismatch: got QUANT_ANALYST expected JPM_STARTUP_BANKER |
| Financial: Semiconductor Market Share | ❌ FAIL | 120.2 | persona mismatch: got QUANT_ANALYST expected ECOSYSTEM_PARTNER |
| Security: CVE-2024-21887 Scoring | ❌ FAIL | 146.7 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Security: CVE-2024-23897 Scoring | ❌ FAIL | 141.2 | unknown ground truth entity: CVE-2024-23897 |
| Security: HTTP/2 Rapid Reset | ❌ FAIL | 132.0 | persona mismatch: got QUANT_ANALYST expected JPM_STARTUP_BANKER |
| Security: Windows SmartScreen CVE | ❌ FAIL | 63.4 | Missing [DEBRIEF_V1_JSON] block |
| Security: FortiOS CVE | ❌ FAIL | 4.7 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Security: Critical Vulnerability Scoring | ❌ FAIL | 127.9 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Security: SQL Injection CVE | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Security: RCE Vulnerability | ❌ FAIL | 163.9 | unknown ground truth entity: CVE-2024-11111 |
| Security: Authentication Bypass | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Security: Path Traversal CVE | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Security: CVE-2024-21887 Remediation | ❌ FAIL | 115.8 | unknown ground truth entity: CVE-2024-21887 |
| Security: CVE-2023-36025 Remediation | ❌ FAIL | 144.9 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Security: Jenkins CVE Remediation | ❌ FAIL | 167.4 | unknown ground truth entity: CVE-2024-23897 |
| Security: HTTP/2 Rapid Reset Mitigation | ❌ FAIL | 114.9 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Security: FortiOS Patching | ❌ FAIL | 139.0 | persona mismatch: got CTO_TECH_LEAD expected ENTERPRISE_EXEC |
| Security: Critical RCE Remediation | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Security: SQL Injection Response | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Security: Auth Bypass Remediation | ❌ FAIL | 16.1 | Missing [DEBRIEF_V1_JSON] block |
| Security: Path Traversal Mitigation | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Security: Supply Chain Response | ❌ FAIL | 74.9 | Missing [DEBRIEF_V1_JSON] block |
| Security: FortiOS Impact Analysis | ❌ FAIL | 156.3 | persona mismatch: got CTO_TECH_LEAD expected ENTERPRISE_EXEC |
| Security: Ransomware Impact Assessment | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Security: Data Breach Analysis | ❌ FAIL | 5.0 | Missing [DEBRIEF_V1_JSON] block |
| Security: Cloud Vulnerability Impact | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Security: Third-Party Risk | ❌ FAIL | 98.3 | persona mismatch: got CTO_TECH_LEAD expected ECOSYSTEM_PARTNER |
| Academic: CRISPR Paper Summary | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Academic: mRNA Vaccine Paper | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| offset:57 | ❌ FAIL | 300.7 | - |
| Academic: Alzheimer's Mechanism | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Cancer Immunotherapy | ❌ FAIL | 5.0 | Missing [DEBRIEF_V1_JSON] block |
| Academic: PD-L1 Research | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Gene Therapy Advances | ❌ FAIL | 111.9 | persona mismatch: got EARLY_STAGE_VC expected JPM_STARTUP_BANKER |
| Academic: Neurodegeneration Research | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Stem Cell Therapy | ❌ FAIL | 121.3 | Uncaught Error: Uncaught Error: Failed after 3 attempts. Last error: This reques |
| Academic: Microbiome Research | ❌ FAIL | 81.7 | unknown ground truth entity: gut microbiome immune response |
| Academic: Protein Folding | ❌ FAIL | 131.6 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Academic: Long COVID Research | ❌ FAIL | 4.7 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Academic: Senolytics Research | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Cardiomyopathy Treatment | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Antibiotic Resistance | ❌ FAIL | 4.9 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Academic: CRISPR Methodology | ❌ FAIL | 104.1 | persona.inferred must be a known persona |
| Academic: Clinical Trial Design | ❌ FAIL | 116.7 | unknown ground truth entity: cancer immunotherapy checkpoint |
| Academic: Animal Model Validation | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Biomarker Discovery | ❌ FAIL | 96.5 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Academic: Genomic Analysis | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Medical Imaging Analysis | ❌ FAIL | 94.2 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Academic: Meta-Analysis Quality | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Academic: RNA-Seq Validation | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Flow Cytometry Standards | ❌ FAIL | 63.6 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Replication Study | ❌ FAIL | 95.9 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Market: Trending AI JavaScript Repos | ❌ FAIL | 129.1 | persona mismatch: got QUANT_ANALYST expected FOUNDER_STRATEGY |
| Market: Trending Python AI Repos | ❌ FAIL | 180.1 | persona mismatch: got QUANT_ANALYST expected JPM_STARTUP_BANKER |
| Market: Trending Rust Systems Repos | ❌ FAIL | 144.3 | persona mismatch: got QUANT_ANALYST expected CTO_TECH_LEAD |
| Market: Trending Go Cloud Repos | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Market: Trending TypeScript Frontend | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Market: Trending Runtime Repos | ❌ FAIL | 5.0 | Missing [DEBRIEF_V1_JSON] block |
| Market: Trending Database Repos | ❌ FAIL | 153.3 | persona mismatch: got QUANT_ANALYST expected FOUNDER_STRATEGY |
| Market: Trending DevOps Tools | ❌ FAIL | 158.5 | Missing [DEBRIEF_V1_JSON] block |
| Market: Trending Blockchain Repos | ❌ FAIL | 170.4 | unknown ground truth entity: created:>2025-12-01 blockchain OR cryptocurrency |
| Market: Trending Mobile Dev Repos | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Market: React vs Vue Comparison | ❌ FAIL | 143.5 | persona mismatch: got QUANT_ANALYST expected JPM_STARTUP_BANKER |
| Market: TensorFlow vs PyTorch | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Market: Angular vs Vue Comparison | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Market: Next.js vs Nuxt Comparison | ❌ FAIL | 132.6 | persona mismatch: got QUANT_ANALYST expected CTO_TECH_LEAD |
| Market: Docker vs Podman Comparison | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Market: Container Orchestration Comparison | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Market: PostgreSQL vs MySQL Comparison | ❌ FAIL | 150.7 | unknown ground truth entity: postgresql mysql |
| Market: MongoDB vs PostgreSQL Comparison | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Market: GraphQL vs REST API Frameworks | ❌ FAIL | 4.7 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Market: Electron vs Tauri Comparison | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |

### gpt-5-mini

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| Financial: Tesla 2024 Revenue | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Apple 2024 Revenue | ❌ FAIL | 75.3 | unknown ground truth entity: 0000320193 |
| Financial: Microsoft 2024 Revenue | ❌ FAIL | 64.3 | persona mismatch: got ENTERPRISE_EXEC expected JPM_STARTUP_BANKER |
| Financial: Alphabet 2024 Revenue | ❌ FAIL | 60.3 | persona mismatch: got QUANT_ANALYST expected EARLY_STAGE_VC |
| Financial: NVIDIA 2024 Revenue | ❌ FAIL | 57.0 | persona mismatch: got QUANT_ANALYST expected FOUNDER_STRATEGY |
| Financial: Amazon 2024 Revenue | ❌ FAIL | 145.3 | persona mismatch: got ENTERPRISE_EXEC expected JPM_STARTUP_BANKER |
| Financial: Meta 2024 Revenue | ❌ FAIL | 164.5 | persona mismatch: got QUANT_ANALYST expected ENTERPRISE_EXEC |
| Financial: AMD 2024 Revenue | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Intel 2024 Revenue | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Qualcomm 2024 Revenue | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Google 2024 Risk Factors | ❌ FAIL | 70.5 | unknown ground truth entity: 0001652044 |
| Financial: NVIDIA 2024 Risk Factors | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Apple 2024 Risk Factors | ❌ FAIL | 4.9 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Financial: Microsoft 2024 Risk Factors | ❌ FAIL | 74.2 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Financial: Tesla 2024 Risk Factors | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Amazon 2024 Risk Factors | ❌ FAIL | 68.2 | persona mismatch: got ENTERPRISE_EXEC expected ECOSYSTEM_PARTNER |
| Financial: Meta 2024 Risk Factors | ❌ FAIL | 107.6 | persona mismatch: got ACADEMIC_RD expected FOUNDER_STRATEGY |
| Financial: AMD 2024 Risk Factors | ❌ FAIL | 154.8 | persona mismatch: got EARLY_STAGE_VC expected JPM_STARTUP_BANKER |
| Financial: Intel 2024 Risk Factors | ❌ FAIL | 4.8 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Financial: Qualcomm 2024 Risk Factors | ❌ FAIL | 224.6 | persona mismatch: got ENTERPRISE_EXEC expected SALES_ENGINEER |
| Financial: AMD Business Description | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: NVIDIA Business Description | ❌ FAIL | 71.4 | persona mismatch: got ENTERPRISE_EXEC expected FOUNDER_STRATEGY |
| Financial: Cloud Business Models | ❌ FAIL | 78.6 | unknown ground truth entity: cloud_providers |
| Financial: Semiconductor Business Models | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Social Media Business Models | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Cloud Market Comparison | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: AI Chip Market | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Social Media Comparison | ❌ FAIL | 108.9 | persona mismatch: got QUANT_ANALYST expected FOUNDER_STRATEGY |
| Financial: Cloud Growth Rates | ❌ FAIL | 52.7 | persona mismatch: got QUANT_ANALYST expected JPM_STARTUP_BANKER |
| Financial: Semiconductor Market Share | ❌ FAIL | 4.9 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Security: CVE-2024-21887 Scoring | ❌ FAIL | 4.7 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Security: CVE-2024-23897 Scoring | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Security: HTTP/2 Rapid Reset | ❌ FAIL | 76.8 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Security: Windows SmartScreen CVE | ❌ FAIL | 132.0 | persona mismatch: got CTO_TECH_LEAD expected ENTERPRISE_EXEC |
| Security: FortiOS CVE | ❌ FAIL | 73.1 | unknown ground truth entity: CVE-2023-27997 |
| Security: Critical Vulnerability Scoring | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Security: SQL Injection CVE | ❌ FAIL | 67.2 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Security: RCE Vulnerability | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Security: Authentication Bypass | ❌ FAIL | 60.2 | persona mismatch: got CTO_TECH_LEAD expected ENTERPRISE_EXEC |
| Security: Path Traversal CVE | ❌ FAIL | 65.3 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Security: CVE-2024-21887 Remediation | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Security: CVE-2023-36025 Remediation | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Security: Jenkins CVE Remediation | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Security: HTTP/2 Rapid Reset Mitigation | ❌ FAIL | 89.2 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Security: FortiOS Patching | ❌ FAIL | 75.1 | persona mismatch: got CTO_TECH_LEAD expected ENTERPRISE_EXEC |
| Security: Critical RCE Remediation | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Security: SQL Injection Response | ❌ FAIL | 75.1 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Security: Auth Bypass Remediation | ❌ FAIL | 67.7 | persona mismatch: got CTO_TECH_LEAD expected ENTERPRISE_EXEC |
| Security: Path Traversal Mitigation | ❌ FAIL | 4.8 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Security: Supply Chain Response | ❌ FAIL | 80.6 | unknown ground truth entity: CVE-2024-11111 |
| Security: FortiOS Impact Analysis | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Security: Ransomware Impact Assessment | ❌ FAIL | 65.8 | persona mismatch: got CTO_TECH_LEAD expected ENTERPRISE_EXEC |
| Security: Data Breach Analysis | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Security: Cloud Vulnerability Impact | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| offset:54 | ❌ FAIL | 300.8 | - |
| Academic: CRISPR Paper Summary | ❌ FAIL | 109.4 | unknown ground truth entity: CRISPR gene editing 2024 |
| Academic: mRNA Vaccine Paper | ❌ FAIL | 62.2 | unknown ground truth entity: mRNA vaccine technology |
| Academic: CAR-T Therapy Paper | ❌ FAIL | 99.1 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Academic: Alzheimer's Mechanism | ❌ FAIL | 59.9 | unknown ground truth entity: Alzheimer's disease mechanism |
| Academic: Cancer Immunotherapy | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Academic: PD-L1 Research | ❌ FAIL | 99.2 | unknown ground truth entity: PD-L1 cancer immunotherapy |
| Academic: Gene Therapy Advances | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Neurodegeneration Research | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Stem Cell Therapy | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Microbiome Research | ❌ FAIL | 75.5 | unknown ground truth entity: gut microbiome immune response |
| Academic: Protein Folding | ❌ FAIL | 88.9 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Academic: Long COVID Research | ❌ FAIL | 107.4 | unknown ground truth entity: long COVID pathophysiology |
| Academic: Senolytics Research | ❌ FAIL | 265.9 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Academic: Cardiomyopathy Treatment | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Antibiotic Resistance | ❌ FAIL | 117.0 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Academic: CRISPR Methodology | ❌ FAIL | 44.7 | unknown ground truth entity: CRISPR gene editing 2024 |
| Academic: Clinical Trial Design | ❌ FAIL | 54.4 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Academic: Animal Model Validation | ❌ FAIL | 84.1 | unknown ground truth entity: Alzheimer's disease mechanism |
| Academic: Biomarker Discovery | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Genomic Analysis | ❌ FAIL | 69.6 | unknown ground truth entity: CRISPR gene editing 2024 |
| Academic: Medical Imaging Analysis | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Meta-Analysis Quality | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Academic: RNA-Seq Validation | ❌ FAIL | 64.1 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Academic: Flow Cytometry Standards | ❌ FAIL | 50.4 | unknown ground truth entity: immune response flow cytometry |
| Academic: Replication Study | ❌ FAIL | 4.7 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Market: Trending AI JavaScript Repos | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Market: Trending Python AI Repos | ❌ FAIL | 149.7 | persona mismatch: got ECOSYSTEM_PARTNER expected JPM_STARTUP_BANKER |
| Market: Trending Rust Systems Repos | ❌ FAIL | 83.1 | persona mismatch: got QUANT_ANALYST expected CTO_TECH_LEAD |
| Market: Trending Go Cloud Repos | ❌ FAIL | 114.6 | persona mismatch: got ECOSYSTEM_PARTNER expected JPM_STARTUP_BANKER |
| Market: Trending TypeScript Frontend | ❌ FAIL | 83.1 | persona mismatch: got ECOSYSTEM_PARTNER expected PRODUCT_DESIGNER |
| Market: Trending Runtime Repos | ❌ FAIL | 214.3 | persona mismatch: got QUANT_ANALYST expected CTO_TECH_LEAD |
| Market: Trending Database Repos | ❌ FAIL | 165.9 | persona mismatch: got ECOSYSTEM_PARTNER expected FOUNDER_STRATEGY |
| Market: Trending DevOps Tools | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Market: Trending Blockchain Repos | ❌ FAIL | 104.8 | persona mismatch: got ECOSYSTEM_PARTNER expected QUANT_ANALYST |
| Market: Trending Mobile Dev Repos | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Market: React vs Vue Comparison | ❌ FAIL | 52.6 | persona mismatch: got QUANT_ANALYST expected JPM_STARTUP_BANKER |
| Market: TensorFlow vs PyTorch | ❌ FAIL | 192.4 | persona mismatch: got ECOSYSTEM_PARTNER expected FOUNDER_STRATEGY |
| Market: Angular vs Vue Comparison | ❌ FAIL | 79.7 | persona mismatch: got CTO_TECH_LEAD expected PRODUCT_DESIGNER |
| Market: Next.js vs Nuxt Comparison | ❌ FAIL | 191.4 | persona mismatch: got QUANT_ANALYST expected CTO_TECH_LEAD |
| Market: Docker vs Podman Comparison | ❌ FAIL | 228.2 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Market: Container Orchestration Comparison | ❌ FAIL | 108.2 | persona mismatch: got QUANT_ANALYST expected ENTERPRISE_EXEC |
| Market: PostgreSQL vs MySQL Comparison | ❌ FAIL | 107.5 | unknown ground truth entity: postgresql mysql |
| Market: MongoDB vs PostgreSQL Comparison | ❌ FAIL | 86.2 | persona mismatch: got CTO_TECH_LEAD expected FOUNDER_STRATEGY |
| Market: GraphQL vs REST API Frameworks | ❌ FAIL | 74.4 | persona mismatch: got QUANT_ANALYST expected JPM_STARTUP_BANKER |
| Market: Electron vs Tauri Comparison | ❌ FAIL | 215.2 | persona mismatch: got CTO_TECH_LEAD expected PRODUCT_DESIGNER |

### gemini-3-flash

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| Financial: Tesla 2024 Revenue | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Apple 2024 Revenue | ❌ FAIL | 94.9 | persona mismatch: got QUANT_ANALYST expected ENTERPRISE_EXEC |
| Financial: Microsoft 2024 Revenue | ❌ FAIL | 11.6 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Alphabet 2024 Revenue | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: NVIDIA 2024 Revenue | ❌ FAIL | 17.0 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Amazon 2024 Revenue | ❌ FAIL | 17.4 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Meta 2024 Revenue | ❌ FAIL | 17.6 | Missing [DEBRIEF_V1_JSON] block |
| Financial: AMD 2024 Revenue | ❌ FAIL | 97.2 | unknown ground truth entity: 0000002488 |
| Financial: Intel 2024 Revenue | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Qualcomm 2024 Revenue | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Google 2024 Risk Factors | ❌ FAIL | 99.7 | unknown ground truth entity: 0001652044 |
| Financial: NVIDIA 2024 Risk Factors | ❌ FAIL | 4.8 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Financial: Apple 2024 Risk Factors | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Microsoft 2024 Risk Factors | ❌ FAIL | 4.7 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Financial: Tesla 2024 Risk Factors | ❌ FAIL | 72.2 | persona mismatch: got ENTERPRISE_EXEC expected EARLY_STAGE_VC |
| Financial: Amazon 2024 Risk Factors | ❌ FAIL | 4.6 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Financial: Meta 2024 Risk Factors | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: AMD 2024 Risk Factors | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Intel 2024 Risk Factors | ❌ FAIL | 15.4 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Qualcomm 2024 Risk Factors | ❌ FAIL | 18.4 | Missing [DEBRIEF_V1_JSON] block |
| Financial: AMD Business Description | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: NVIDIA Business Description | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Cloud Business Models | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Semiconductor Business Models | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Social Media Business Models | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Cloud Market Comparison | ❌ FAIL | 75.6 | persona mismatch: got QUANT_ANALYST expected EARLY_STAGE_VC |
| Financial: AI Chip Market | ❌ FAIL | 4.5 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Social Media Comparison | ❌ FAIL | 18.1 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Cloud Growth Rates | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Semiconductor Market Share | ❌ FAIL | 15.7 | Missing [DEBRIEF_V1_JSON] block |
| Security: CVE-2024-21887 Scoring | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Security: CVE-2024-23897 Scoring | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Security: HTTP/2 Rapid Reset | ❌ FAIL | 20.1 | Missing [DEBRIEF_V1_JSON] block |
| Security: Windows SmartScreen CVE | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Security: FortiOS CVE | ❌ FAIL | 82.2 | unknown ground truth entity: CVE-2023-27997 |
| Security: Critical Vulnerability Scoring | ❌ FAIL | 86.2 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Security: SQL Injection CVE | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Security: RCE Vulnerability | ❌ FAIL | 74.7 | unknown ground truth entity: CVE-2024-11111 |
| Security: Authentication Bypass | ❌ FAIL | 35.5 | Missing [DEBRIEF_V1_JSON] block |
| Security: Path Traversal CVE | ❌ FAIL | 52.5 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Security: CVE-2024-21887 Remediation | ❌ FAIL | 104.2 | unknown ground truth entity: CVE-2024-21887 |
| Security: CVE-2023-36025 Remediation | ❌ FAIL | 57.7 | persona mismatch: got CTO_TECH_LEAD expected JPM_STARTUP_BANKER |
| Security: Jenkins CVE Remediation | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Security: HTTP/2 Rapid Reset Mitigation | ❌ FAIL | 20.2 | Missing [DEBRIEF_V1_JSON] block |
| Security: FortiOS Patching | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Security: Critical RCE Remediation | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Security: SQL Injection Response | ❌ FAIL | 4.6 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Security: Auth Bypass Remediation | ❌ FAIL | 42.3 | persona mismatch: got CTO_TECH_LEAD expected ENTERPRISE_EXEC |
| Security: Path Traversal Mitigation | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Security: Supply Chain Response | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Security: FortiOS Impact Analysis | ❌ FAIL | 70.5 | persona mismatch: got CTO_TECH_LEAD expected ENTERPRISE_EXEC |
| Security: Ransomware Impact Assessment | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Security: Data Breach Analysis | ❌ FAIL | 57.1 | persona mismatch: got ENTERPRISE_EXEC expected QUANT_ANALYST |
| Security: Cloud Vulnerability Impact | ❌ FAIL | 11.5 | Missing [DEBRIEF_V1_JSON] block |
| Security: Third-Party Risk | ❌ FAIL | 17.3 | Missing [DEBRIEF_V1_JSON] block |
| Academic: CRISPR Paper Summary | ❌ FAIL | 4.5 | Missing [DEBRIEF_V1_JSON] block |
| Academic: mRNA Vaccine Paper | ❌ FAIL | 33.5 | Missing [DEBRIEF_V1_JSON] block |
| Academic: CAR-T Therapy Paper | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Alzheimer's Mechanism | ❌ FAIL | 102.1 | unknown ground truth entity: Alzheimer's disease mechanism |
| Academic: Cancer Immunotherapy | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Academic: PD-L1 Research | ❌ FAIL | 16.1 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Gene Therapy Advances | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Neurodegeneration Research | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Stem Cell Therapy | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Microbiome Research | ❌ FAIL | 11.5 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Protein Folding | ❌ FAIL | 16.1 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Long COVID Research | ❌ FAIL | 15.0 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Senolytics Research | ❌ FAIL | 86.2 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Academic: Cardiomyopathy Treatment | ❌ FAIL | 28.7 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Antibiotic Resistance | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Academic: CRISPR Methodology | ❌ FAIL | 4.6 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Clinical Trial Design | ❌ FAIL | 70.5 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Academic: Animal Model Validation | ❌ FAIL | 47.8 | unknown ground truth entity: Alzheimer's disease mechanism |
| Academic: Biomarker Discovery | ❌ FAIL | 11.5 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Genomic Analysis | ❌ FAIL | 4.9 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Academic: Medical Imaging Analysis | ❌ FAIL | 37.8 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Academic: Meta-Analysis Quality | ❌ FAIL | 121.3 | Uncaught Error: Uncaught Error: Failed after 3 attempts. Last error: You exceede |
| Academic: RNA-Seq Validation | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Academic: Flow Cytometry Standards | ❌ FAIL | 69.0 | unknown ground truth entity: immune response flow cytometry |
| Academic: Replication Study | ❌ FAIL | 35.0 | persona mismatch: got ACADEMIC_RD expected JPM_STARTUP_BANKER |
| Market: Trending AI JavaScript Repos | ❌ FAIL | 148.6 | persona mismatch: got QUANT_ANALYST expected FOUNDER_STRATEGY |
| Market: Trending Python AI Repos | ❌ FAIL | 11.5 | Missing [DEBRIEF_V1_JSON] block |
| Market: Trending Rust Systems Repos | ❌ FAIL | 4.8 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Market: Trending Go Cloud Repos | ❌ FAIL | 17.2 | Missing [DEBRIEF_V1_JSON] block |
| Market: Trending TypeScript Frontend | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Market: Trending Runtime Repos | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Market: Trending Database Repos | ❌ FAIL | 134.4 | persona mismatch: got QUANT_ANALYST expected FOUNDER_STRATEGY |
| Market: Trending DevOps Tools | ❌ FAIL | 121.3 | Uncaught Error: Uncaught Error: Failed after 3 attempts. Last error: You exceede |
| Market: Trending Blockchain Repos | ❌ FAIL | 15.5 | Missing [DEBRIEF_V1_JSON] block |
| Market: Trending Mobile Dev Repos | ❌ FAIL | 4.5 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Market: React vs Vue Comparison | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| offset:91 | ❌ ERROR | 10.7 | fetch failed |
| Market: Angular vs Vue Comparison | ❌ FAIL | 102.4 | Missing [DEBRIEF_V1_JSON] block |
| Market: Next.js vs Nuxt Comparison | ❌ FAIL | 84.5 | persona mismatch: got QUANT_ANALYST expected CTO_TECH_LEAD |
| Market: Docker vs Podman Comparison | ❌ FAIL | 16.5 | Missing [DEBRIEF_V1_JSON] block |
| Market: Container Orchestration Comparison | ❌ FAIL | 4.9 | Missing [DEBRIEF_V1_JSON] block |
| Market: PostgreSQL vs MySQL Comparison | ❌ FAIL | 4.7 | Uncaught Error: Too many concurrent commits in a short period of time. Spread yo |
| Market: MongoDB vs PostgreSQL Comparison | ❌ FAIL | 79.5 | persona mismatch: got EARLY_STAGE_VC expected FOUNDER_STRATEGY |
| offset:98 | ❌ ERROR | 10.7 | fetch failed |
| Market: Electron vs Tauri Comparison | ❌ FAIL | 56.4 | persona mismatch: got QUANT_ANALYST expected PRODUCT_DESIGNER |

### deepseek-r1

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| Financial: Tesla 2024 Revenue | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| offset:1 | ❌ FAIL | 301.0 | - |
| Financial: Microsoft 2024 Revenue | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Alphabet 2024 Revenue | ❌ FAIL | 263.1 | Missing [DEBRIEF_V1_JSON] block |
| Financial: NVIDIA 2024 Revenue | ❌ FAIL | 283.9 | Missing [DEBRIEF_V1_JSON] block |
| offset:5 | ❌ ERROR | 10.7 | fetch failed |
| Financial: Meta 2024 Revenue | ❌ FAIL | 149.2 | Missing [DEBRIEF_V1_JSON] block |
| Financial: AMD 2024 Revenue | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Intel 2024 Revenue | ❌ FAIL | 169.4 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Qualcomm 2024 Revenue | ❌ FAIL | 227.1 | persona mismatch: got QUANT_ANALYST expected JPM_STARTUP_BANKER |
| offset:10 | ❌ ERROR | 10.7 | fetch failed |
| offset:11 | ❌ FAIL | 301.1 | - |
| Financial: Apple 2024 Risk Factors | ❌ FAIL | 89.2 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Microsoft 2024 Risk Factors | ❌ FAIL | 89.2 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Tesla 2024 Risk Factors | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Amazon 2024 Risk Factors | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| offset:16 | ❌ FAIL | 301.1 | - |
| Financial: AMD 2024 Risk Factors | ❌ FAIL | 88.9 | Missing [DEBRIEF_V1_JSON] block |
| offset:18 | ❌ FAIL | 301.1 | - |
| offset:19 | ❌ ERROR | 10.7 | fetch failed |
| Financial: AMD Business Description | ❌ FAIL | 288.5 | Missing [DEBRIEF_V1_JSON] block |
| offset:21 | ❌ ERROR | 10.7 | fetch failed |
| offset:22 | ❌ ERROR | 10.7 | fetch failed |
| Financial: Semiconductor Business Models | ❌ FAIL | 4.7 | Missing [DEBRIEF_V1_JSON] block |
| Financial: Social Media Business Models | ❌ FAIL | 4.8 | Missing [DEBRIEF_V1_JSON] block |
| offset:25 | ❌ ERROR | 10.7 | fetch failed |
| offset:26 | ❌ ERROR | 10.7 | fetch failed |
| offset:27 | ❌ ERROR | 10.7 | fetch failed |
| offset:28 | ❌ ERROR | 10.7 | fetch failed |
| offset:29 | ❌ ERROR | 10.7 | fetch failed |
| offset:30 | ❌ ERROR | 10.7 | fetch failed |
| offset:31 | ❌ ERROR | 10.7 | fetch failed |
| offset:32 | ❌ ERROR | 10.7 | fetch failed |
| offset:33 | ❌ ERROR | 10.7 | fetch failed |
| offset:34 | ❌ ERROR | 10.7 | fetch failed |
| offset:35 | ❌ ERROR | 10.7 | fetch failed |
| offset:36 | ❌ ERROR | 10.7 | fetch failed |
| offset:37 | ❌ ERROR | 10.7 | fetch failed |
| offset:38 | ❌ ERROR | 10.7 | fetch failed |
| offset:39 | ❌ ERROR | 10.7 | fetch failed |
| offset:40 | ❌ ERROR | 10.7 | fetch failed |
| offset:41 | ❌ ERROR | 10.7 | fetch failed |
| offset:42 | ❌ ERROR | 10.7 | fetch failed |
| offset:43 | ❌ ERROR | 10.7 | fetch failed |
| offset:44 | ❌ ERROR | 10.7 | fetch failed |
| offset:45 | ❌ ERROR | 10.7 | fetch failed |
| offset:46 | ❌ ERROR | 10.7 | fetch failed |
| offset:47 | ❌ ERROR | 10.7 | fetch failed |
| offset:48 | ❌ ERROR | 10.7 | fetch failed |
| offset:49 | ❌ ERROR | 10.7 | fetch failed |
| offset:50 | ❌ ERROR | 10.7 | fetch failed |
| offset:51 | ❌ ERROR | 10.7 | fetch failed |
| offset:52 | ❌ ERROR | 10.7 | fetch failed |
| offset:53 | ❌ ERROR | 10.7 | fetch failed |
| offset:54 | ❌ ERROR | 10.7 | fetch failed |
| offset:55 | ❌ ERROR | 10.7 | fetch failed |
| offset:56 | ❌ ERROR | 10.7 | fetch failed |
| offset:57 | ❌ ERROR | 10.7 | fetch failed |
| offset:58 | ❌ ERROR | 10.7 | fetch failed |
| offset:59 | ❌ ERROR | 10.7 | fetch failed |
| offset:60 | ❌ ERROR | 10.7 | fetch failed |
| offset:61 | ❌ ERROR | 10.7 | fetch failed |
| offset:62 | ❌ ERROR | 10.7 | fetch failed |
| offset:63 | ❌ ERROR | 10.7 | fetch failed |
| offset:64 | ❌ ERROR | 10.7 | fetch failed |
| offset:65 | ❌ ERROR | 10.7 | fetch failed |
| offset:66 | ❌ ERROR | 10.7 | fetch failed |
| offset:67 | ❌ ERROR | 10.7 | fetch failed |
| offset:68 | ❌ ERROR | 10.7 | fetch failed |
| offset:69 | ❌ ERROR | 10.7 | fetch failed |
| offset:70 | ❌ ERROR | 10.7 | fetch failed |
| offset:71 | ❌ ERROR | 10.7 | fetch failed |
| offset:72 | ❌ ERROR | 10.7 | fetch failed |
| offset:73 | ❌ ERROR | 10.7 | fetch failed |
| offset:74 | ❌ ERROR | 10.7 | fetch failed |
| offset:75 | ❌ ERROR | 10.7 | fetch failed |
| offset:76 | ❌ ERROR | 10.7 | fetch failed |
| offset:77 | ❌ ERROR | 10.7 | fetch failed |
| offset:78 | ❌ ERROR | 10.7 | fetch failed |
| offset:79 | ❌ ERROR | 10.7 | fetch failed |
| offset:80 | ❌ ERROR | 10.7 | fetch failed |
| offset:81 | ❌ ERROR | 10.7 | fetch failed |
| offset:82 | ❌ ERROR | 10.7 | fetch failed |
| offset:83 | ❌ ERROR | 10.7 | fetch failed |
| offset:84 | ❌ ERROR | 10.7 | fetch failed |
| offset:85 | ❌ ERROR | 10.7 | fetch failed |
| offset:86 | ❌ ERROR | 10.7 | fetch failed |
| offset:87 | ❌ ERROR | 10.7 | fetch failed |
| offset:88 | ❌ ERROR | 10.7 | fetch failed |
| offset:89 | ❌ ERROR | 10.7 | fetch failed |
| offset:90 | ❌ ERROR | 10.7 | fetch failed |
| offset:91 | ❌ ERROR | 10.7 | fetch failed |
| offset:92 | ❌ ERROR | 10.7 | fetch failed |
| offset:93 | ❌ ERROR | 10.7 | fetch failed |
| offset:94 | ❌ ERROR | 10.7 | fetch failed |
| offset:95 | ❌ ERROR | 10.7 | fetch failed |
| offset:96 | ❌ ERROR | 10.7 | fetch failed |
| offset:97 | ❌ ERROR | 10.7 | fetch failed |
| offset:98 | ❌ ERROR | 10.7 | fetch failed |
| offset:99 | ❌ ERROR | 10.7 | fetch failed |

### deepseek-v3.2

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| offset:0 | ❌ ERROR | 10.7 | fetch failed |
| offset:1 | ❌ ERROR | 10.7 | fetch failed |
| offset:2 | ❌ ERROR | 10.7 | fetch failed |
| offset:3 | ❌ ERROR | 10.7 | fetch failed |
| offset:4 | ❌ ERROR | 10.7 | fetch failed |
| offset:5 | ❌ ERROR | 10.7 | fetch failed |
| offset:6 | ❌ ERROR | 10.7 | fetch failed |
| offset:7 | ❌ ERROR | 10.7 | fetch failed |
| offset:8 | ❌ ERROR | 10.7 | fetch failed |
| offset:9 | ❌ ERROR | 10.7 | fetch failed |
| offset:10 | ❌ ERROR | 10.7 | fetch failed |
| offset:11 | ❌ ERROR | 10.7 | fetch failed |
| offset:12 | ❌ ERROR | 10.7 | fetch failed |
| offset:13 | ❌ ERROR | 10.7 | fetch failed |
| offset:14 | ❌ ERROR | 10.7 | fetch failed |
| offset:15 | ❌ ERROR | 10.7 | fetch failed |
| offset:16 | ❌ ERROR | 10.7 | fetch failed |
| offset:17 | ❌ ERROR | 10.7 | fetch failed |
| offset:18 | ❌ ERROR | 10.7 | fetch failed |
| offset:19 | ❌ ERROR | 10.7 | fetch failed |
| offset:20 | ❌ ERROR | 10.7 | fetch failed |
| offset:21 | ❌ ERROR | 10.7 | fetch failed |
| offset:22 | ❌ ERROR | 10.7 | fetch failed |
| offset:23 | ❌ ERROR | 10.7 | fetch failed |
| offset:24 | ❌ ERROR | 10.7 | fetch failed |
| offset:25 | ❌ ERROR | 10.7 | fetch failed |
| offset:26 | ❌ ERROR | 10.7 | fetch failed |
| offset:27 | ❌ ERROR | 10.7 | fetch failed |
| offset:28 | ❌ ERROR | 10.7 | fetch failed |
| offset:29 | ❌ ERROR | 10.7 | fetch failed |
| offset:30 | ❌ ERROR | 10.7 | fetch failed |
| offset:31 | ❌ ERROR | 10.7 | fetch failed |
| offset:32 | ❌ ERROR | 10.7 | fetch failed |
| offset:33 | ❌ ERROR | 10.7 | fetch failed |
| offset:34 | ❌ ERROR | 10.7 | fetch failed |
| offset:35 | ❌ ERROR | 10.7 | fetch failed |
| offset:36 | ❌ ERROR | 10.7 | fetch failed |
| offset:37 | ❌ ERROR | 10.7 | fetch failed |
| offset:38 | ❌ ERROR | 10.7 | fetch failed |
| offset:39 | ❌ ERROR | 10.7 | fetch failed |
| offset:40 | ❌ ERROR | 10.7 | fetch failed |
| offset:41 | ❌ ERROR | 10.7 | fetch failed |
| offset:42 | ❌ ERROR | 10.7 | fetch failed |
| offset:43 | ❌ ERROR | 10.7 | fetch failed |
| offset:44 | ❌ ERROR | 10.7 | fetch failed |
| offset:45 | ❌ ERROR | 10.7 | fetch failed |
| offset:46 | ❌ ERROR | 10.7 | fetch failed |
| offset:47 | ❌ ERROR | 10.7 | fetch failed |
| offset:48 | ❌ ERROR | 10.7 | fetch failed |
| offset:49 | ❌ ERROR | 10.7 | fetch failed |
| offset:50 | ❌ ERROR | 10.7 | fetch failed |
| offset:51 | ❌ ERROR | 10.7 | fetch failed |
| offset:52 | ❌ ERROR | 10.7 | fetch failed |
| offset:53 | ❌ ERROR | 10.7 | fetch failed |
| offset:54 | ❌ ERROR | 10.7 | fetch failed |
| offset:55 | ❌ ERROR | 10.7 | fetch failed |
| offset:56 | ❌ ERROR | 10.7 | fetch failed |
| offset:57 | ❌ ERROR | 10.7 | fetch failed |
| offset:58 | ❌ ERROR | 10.7 | fetch failed |
| offset:59 | ❌ ERROR | 10.7 | fetch failed |
| offset:60 | ❌ ERROR | 10.7 | fetch failed |
| offset:61 | ❌ ERROR | 10.7 | fetch failed |
| offset:62 | ❌ ERROR | 10.7 | fetch failed |
| offset:63 | ❌ ERROR | 10.7 | fetch failed |
| offset:64 | ❌ ERROR | 10.7 | fetch failed |
| offset:65 | ❌ ERROR | 10.7 | fetch failed |
| offset:66 | ❌ ERROR | 10.7 | fetch failed |
| offset:67 | ❌ ERROR | 10.7 | fetch failed |
| offset:68 | ❌ ERROR | 10.7 | fetch failed |
| offset:69 | ❌ ERROR | 10.7 | fetch failed |
| offset:70 | ❌ ERROR | 10.7 | fetch failed |
| offset:71 | ❌ ERROR | 10.7 | fetch failed |
| offset:72 | ❌ ERROR | 10.7 | fetch failed |
| offset:73 | ❌ ERROR | 10.7 | fetch failed |
| offset:74 | ❌ ERROR | 10.7 | fetch failed |
| offset:75 | ❌ ERROR | 10.7 | fetch failed |
| offset:76 | ❌ ERROR | 10.7 | fetch failed |
| offset:77 | ❌ ERROR | 10.6 | fetch failed |
| offset:78 | ❌ ERROR | 10.6 | fetch failed |
| offset:79 | ❌ ERROR | 10.6 | fetch failed |
| offset:80 | ❌ ERROR | 10.6 | fetch failed |
| offset:81 | ❌ ERROR | 10.6 | fetch failed |
| offset:82 | ❌ ERROR | 10.6 | fetch failed |
| offset:83 | ❌ ERROR | 10.6 | fetch failed |
| offset:84 | ❌ ERROR | 10.6 | fetch failed |
| offset:85 | ❌ ERROR | 10.6 | fetch failed |
| offset:86 | ❌ ERROR | 10.6 | fetch failed |
| offset:87 | ❌ ERROR | 10.6 | fetch failed |
| offset:88 | ❌ ERROR | 10.6 | fetch failed |
| offset:89 | ❌ ERROR | 10.6 | fetch failed |
| offset:90 | ❌ ERROR | 10.6 | fetch failed |
| offset:91 | ❌ ERROR | 10.6 | fetch failed |
| offset:92 | ❌ ERROR | 10.6 | fetch failed |
| offset:93 | ❌ ERROR | 10.6 | fetch failed |
| offset:94 | ❌ ERROR | 10.6 | fetch failed |
| offset:95 | ❌ ERROR | 10.6 | fetch failed |
| offset:96 | ❌ ERROR | 10.6 | fetch failed |
| offset:97 | ❌ ERROR | 10.6 | fetch failed |
| offset:98 | ❌ ERROR | 10.6 | fetch failed |
| offset:99 | ❌ ERROR | 10.6 | fetch failed |

### qwen3-235b

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| offset:0 | ❌ ERROR | 10.6 | fetch failed |
| offset:1 | ❌ ERROR | 10.6 | fetch failed |
| offset:2 | ❌ ERROR | 10.6 | fetch failed |
| offset:3 | ❌ ERROR | 10.6 | fetch failed |
| offset:4 | ❌ ERROR | 10.6 | fetch failed |
| offset:5 | ❌ ERROR | 10.6 | fetch failed |
| offset:6 | ❌ ERROR | 10.6 | fetch failed |
| offset:7 | ❌ ERROR | 10.6 | fetch failed |
| offset:8 | ❌ ERROR | 10.6 | fetch failed |
| offset:9 | ❌ ERROR | 10.6 | fetch failed |
| offset:10 | ❌ ERROR | 10.6 | fetch failed |
| offset:11 | ❌ ERROR | 10.6 | fetch failed |
| offset:12 | ❌ ERROR | 10.6 | fetch failed |
| offset:13 | ❌ ERROR | 10.6 | fetch failed |
| offset:14 | ❌ ERROR | 10.6 | fetch failed |
| offset:15 | ❌ ERROR | 10.6 | fetch failed |
| offset:16 | ❌ ERROR | 10.6 | fetch failed |
| offset:17 | ❌ ERROR | 10.6 | fetch failed |
| offset:18 | ❌ ERROR | 10.6 | fetch failed |
| offset:19 | ❌ ERROR | 10.6 | fetch failed |
| offset:20 | ❌ ERROR | 10.6 | fetch failed |
| offset:21 | ❌ ERROR | 10.6 | fetch failed |
| offset:22 | ❌ ERROR | 10.6 | fetch failed |
| offset:23 | ❌ ERROR | 10.6 | fetch failed |
| offset:24 | ❌ ERROR | 10.6 | fetch failed |
| offset:25 | ❌ ERROR | 10.6 | fetch failed |
| offset:26 | ❌ ERROR | 10.6 | fetch failed |
| offset:27 | ❌ ERROR | 10.6 | fetch failed |
| offset:28 | ❌ ERROR | 10.6 | fetch failed |
| offset:29 | ❌ ERROR | 10.6 | fetch failed |
| offset:30 | ❌ ERROR | 10.6 | fetch failed |
| offset:31 | ❌ ERROR | 10.6 | fetch failed |
| offset:32 | ❌ ERROR | 10.6 | fetch failed |
| offset:33 | ❌ ERROR | 10.6 | fetch failed |
| offset:34 | ❌ ERROR | 10.6 | fetch failed |
| offset:35 | ❌ ERROR | 10.6 | fetch failed |
| offset:36 | ❌ ERROR | 10.6 | fetch failed |
| offset:37 | ❌ ERROR | 10.6 | fetch failed |
| offset:38 | ❌ ERROR | 10.6 | fetch failed |
| offset:39 | ❌ ERROR | 10.6 | fetch failed |
| offset:40 | ❌ ERROR | 10.6 | fetch failed |
| offset:41 | ❌ ERROR | 10.6 | fetch failed |
| offset:42 | ❌ ERROR | 10.6 | fetch failed |
| offset:43 | ❌ ERROR | 10.6 | fetch failed |
| offset:44 | ❌ ERROR | 10.6 | fetch failed |
| offset:45 | ❌ ERROR | 10.6 | fetch failed |
| offset:46 | ❌ ERROR | 10.6 | fetch failed |
| offset:47 | ❌ ERROR | 10.6 | fetch failed |
| offset:48 | ❌ ERROR | 10.6 | fetch failed |
| offset:49 | ❌ ERROR | 10.6 | fetch failed |
| offset:50 | ❌ ERROR | 10.6 | fetch failed |
| offset:51 | ❌ ERROR | 10.6 | fetch failed |
| offset:52 | ❌ ERROR | 10.6 | fetch failed |
| offset:53 | ❌ ERROR | 10.6 | fetch failed |
| offset:54 | ❌ ERROR | 10.6 | fetch failed |
| offset:55 | ❌ ERROR | 10.6 | fetch failed |
| offset:56 | ❌ ERROR | 10.6 | fetch failed |
| offset:57 | ❌ ERROR | 10.6 | fetch failed |
| offset:58 | ❌ ERROR | 10.6 | fetch failed |
| offset:59 | ❌ ERROR | 10.6 | fetch failed |
| offset:60 | ❌ ERROR | 10.6 | fetch failed |
| offset:61 | ❌ ERROR | 10.6 | fetch failed |
| offset:62 | ❌ ERROR | 10.6 | fetch failed |
| offset:63 | ❌ ERROR | 10.6 | fetch failed |
| offset:64 | ❌ ERROR | 10.6 | fetch failed |
| offset:65 | ❌ ERROR | 10.6 | fetch failed |
| offset:66 | ❌ ERROR | 10.6 | fetch failed |
| offset:67 | ❌ ERROR | 10.6 | fetch failed |
| offset:68 | ❌ ERROR | 10.6 | fetch failed |
| offset:69 | ❌ ERROR | 10.6 | fetch failed |
| offset:70 | ❌ ERROR | 10.6 | fetch failed |
| offset:71 | ❌ ERROR | 10.6 | fetch failed |
| offset:72 | ❌ ERROR | 10.6 | fetch failed |
| offset:73 | ❌ ERROR | 10.6 | fetch failed |
| offset:74 | ❌ ERROR | 10.6 | fetch failed |
| offset:75 | ❌ ERROR | 10.6 | fetch failed |
| offset:76 | ❌ ERROR | 10.6 | fetch failed |
| offset:77 | ❌ ERROR | 10.6 | fetch failed |
| offset:78 | ❌ ERROR | 10.6 | fetch failed |
| offset:79 | ❌ ERROR | 10.6 | fetch failed |
| offset:80 | ❌ ERROR | 10.6 | fetch failed |
| offset:81 | ❌ ERROR | 10.6 | fetch failed |
| offset:82 | ❌ ERROR | 10.6 | fetch failed |
| offset:83 | ❌ ERROR | 10.6 | fetch failed |
| offset:84 | ❌ ERROR | 10.6 | fetch failed |
| offset:85 | ❌ ERROR | 10.6 | fetch failed |
| offset:86 | ❌ ERROR | 10.6 | fetch failed |
| offset:87 | ❌ ERROR | 10.6 | fetch failed |
| offset:88 | ❌ ERROR | 10.6 | fetch failed |
| offset:89 | ❌ ERROR | 10.6 | fetch failed |
| offset:90 | ❌ ERROR | 10.6 | fetch failed |
| offset:91 | ❌ ERROR | 10.6 | fetch failed |
| offset:92 | ❌ ERROR | 10.6 | fetch failed |
| offset:93 | ❌ ERROR | 10.6 | fetch failed |
| offset:94 | ❌ ERROR | 10.6 | fetch failed |
| offset:95 | ❌ ERROR | 10.6 | fetch failed |
| offset:96 | ❌ ERROR | 10.6 | fetch failed |
| offset:97 | ❌ ERROR | 10.6 | fetch failed |
| offset:98 | ❌ ERROR | 10.6 | fetch failed |
| offset:99 | ❌ ERROR | 10.6 | fetch failed |

### minimax-m2.1

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| offset:0 | ❌ ERROR | 10.6 | fetch failed |
| offset:1 | ❌ ERROR | 10.6 | fetch failed |
| offset:2 | ❌ ERROR | 10.6 | fetch failed |
| offset:3 | ❌ ERROR | 10.6 | fetch failed |
| offset:4 | ❌ ERROR | 10.6 | fetch failed |
| offset:5 | ❌ ERROR | 10.6 | fetch failed |
| offset:6 | ❌ ERROR | 10.6 | fetch failed |
| offset:7 | ❌ ERROR | 10.6 | fetch failed |
| offset:8 | ❌ ERROR | 10.6 | fetch failed |
| offset:9 | ❌ ERROR | 10.6 | fetch failed |
| offset:10 | ❌ ERROR | 10.6 | fetch failed |
| offset:11 | ❌ ERROR | 10.6 | fetch failed |
| offset:12 | ❌ ERROR | 10.6 | fetch failed |
| offset:13 | ❌ ERROR | 10.6 | fetch failed |
| offset:14 | ❌ ERROR | 10.6 | fetch failed |
| offset:15 | ❌ ERROR | 10.6 | fetch failed |
| offset:16 | ❌ ERROR | 10.6 | fetch failed |
| offset:17 | ❌ ERROR | 10.6 | fetch failed |
| offset:18 | ❌ ERROR | 10.6 | fetch failed |
| offset:19 | ❌ ERROR | 10.6 | fetch failed |
| offset:20 | ❌ ERROR | 10.6 | fetch failed |
| offset:21 | ❌ ERROR | 10.6 | fetch failed |
| offset:22 | ❌ ERROR | 10.6 | fetch failed |
| offset:23 | ❌ ERROR | 10.6 | fetch failed |
| offset:24 | ❌ ERROR | 10.6 | fetch failed |
| offset:25 | ❌ ERROR | 10.6 | fetch failed |
| offset:26 | ❌ ERROR | 10.6 | fetch failed |
| offset:27 | ❌ ERROR | 10.6 | fetch failed |
| offset:28 | ❌ ERROR | 10.6 | fetch failed |
| offset:29 | ❌ ERROR | 10.6 | fetch failed |
| offset:30 | ❌ ERROR | 10.6 | fetch failed |
| offset:31 | ❌ ERROR | 10.6 | fetch failed |
| offset:32 | ❌ ERROR | 10.6 | fetch failed |
| offset:33 | ❌ ERROR | 10.6 | fetch failed |
| offset:34 | ❌ ERROR | 10.6 | fetch failed |
| offset:35 | ❌ ERROR | 10.6 | fetch failed |
| offset:36 | ❌ ERROR | 10.6 | fetch failed |
| offset:37 | ❌ ERROR | 10.6 | fetch failed |
| offset:38 | ❌ ERROR | 10.6 | fetch failed |
| offset:39 | ❌ ERROR | 10.6 | fetch failed |
| offset:40 | ❌ ERROR | 10.6 | fetch failed |
| offset:41 | ❌ ERROR | 10.6 | fetch failed |
| offset:42 | ❌ ERROR | 10.6 | fetch failed |
| offset:43 | ❌ ERROR | 10.6 | fetch failed |
| offset:44 | ❌ ERROR | 10.6 | fetch failed |
| offset:45 | ❌ ERROR | 10.6 | fetch failed |
| offset:46 | ❌ ERROR | 10.6 | fetch failed |
| offset:47 | ❌ ERROR | 10.6 | fetch failed |
| offset:48 | ❌ ERROR | 10.6 | fetch failed |
| offset:49 | ❌ ERROR | 10.6 | fetch failed |
| offset:50 | ❌ ERROR | 10.6 | fetch failed |
| offset:51 | ❌ ERROR | 10.6 | fetch failed |
| offset:52 | ❌ ERROR | 10.6 | fetch failed |
| offset:53 | ❌ ERROR | 10.6 | fetch failed |
| offset:54 | ❌ ERROR | 10.6 | fetch failed |
| offset:55 | ❌ ERROR | 10.6 | fetch failed |
| offset:56 | ❌ ERROR | 10.6 | fetch failed |
| offset:57 | ❌ ERROR | 10.6 | fetch failed |
| offset:58 | ❌ ERROR | 10.6 | fetch failed |
| offset:59 | ❌ ERROR | 10.6 | fetch failed |
| offset:60 | ❌ ERROR | 10.6 | fetch failed |
| offset:61 | ❌ ERROR | 10.6 | fetch failed |
| offset:62 | ❌ ERROR | 10.6 | fetch failed |
| offset:63 | ❌ ERROR | 10.6 | fetch failed |
| offset:64 | ❌ ERROR | 10.6 | fetch failed |
| offset:65 | ❌ ERROR | 10.6 | fetch failed |
| offset:66 | ❌ ERROR | 10.6 | fetch failed |
| offset:67 | ❌ ERROR | 10.6 | fetch failed |
| offset:68 | ❌ ERROR | 10.6 | fetch failed |
| offset:69 | ❌ ERROR | 10.6 | fetch failed |
| offset:70 | ❌ ERROR | 10.6 | fetch failed |
| offset:71 | ❌ ERROR | 10.6 | fetch failed |
| offset:72 | ❌ ERROR | 10.6 | fetch failed |
| offset:73 | ❌ ERROR | 10.6 | fetch failed |
| offset:74 | ❌ ERROR | 10.6 | fetch failed |
| offset:75 | ❌ ERROR | 10.6 | fetch failed |
| offset:76 | ❌ ERROR | 10.6 | fetch failed |
| offset:77 | ❌ ERROR | 10.6 | fetch failed |
| offset:78 | ❌ ERROR | 10.6 | fetch failed |
| offset:79 | ❌ ERROR | 10.6 | fetch failed |
| offset:80 | ❌ ERROR | 10.6 | fetch failed |
| offset:81 | ❌ ERROR | 10.6 | fetch failed |
| offset:82 | ❌ ERROR | 10.6 | fetch failed |
| offset:83 | ❌ ERROR | 10.6 | fetch failed |
| offset:84 | ❌ ERROR | 10.6 | fetch failed |
| offset:85 | ❌ ERROR | 10.6 | fetch failed |
| offset:86 | ❌ ERROR | 10.6 | fetch failed |
| offset:87 | ❌ ERROR | 10.6 | fetch failed |
| offset:88 | ❌ ERROR | 10.6 | fetch failed |
| offset:89 | ❌ ERROR | 10.6 | fetch failed |
| offset:90 | ❌ ERROR | 10.6 | fetch failed |
| offset:91 | ❌ ERROR | 10.6 | fetch failed |
| offset:92 | ❌ ERROR | 10.6 | fetch failed |
| offset:93 | ❌ ERROR | 10.6 | fetch failed |
| offset:94 | ❌ ERROR | 10.6 | fetch failed |
| offset:95 | ❌ ERROR | 10.6 | fetch failed |
| offset:96 | ❌ ERROR | 10.6 | fetch failed |
| offset:97 | ❌ ERROR | 10.6 | fetch failed |
| offset:98 | ❌ ERROR | 10.6 | fetch failed |
| offset:99 | ❌ ERROR | 10.6 | fetch failed |

### mimo-v2-flash-free

| Scenario | Status | Time (s) | Failures |
|----------|--------|----------|----------|
| offset:0 | ❌ ERROR | 10.6 | fetch failed |
| Financial: Apple 2024 Revenue | ❌ FAIL | 9.4 | Missing [DEBRIEF_V1_JSON] block |
| offset:2 | ❌ ERROR | 10.6 | fetch failed |
| offset:3 | ❌ ERROR | 10.6 | fetch failed |
| offset:4 | ❌ ERROR | 10.6 | fetch failed |
| offset:5 | ❌ ERROR | 10.6 | fetch failed |
| offset:6 | ❌ ERROR | 10.6 | fetch failed |
| offset:7 | ❌ ERROR | 10.6 | fetch failed |
| offset:8 | ❌ ERROR | 10.6 | fetch failed |
| offset:9 | ❌ ERROR | 10.6 | fetch failed |
| offset:10 | ❌ ERROR | 10.6 | fetch failed |
| offset:11 | ❌ ERROR | 10.6 | fetch failed |
| offset:12 | ❌ ERROR | 10.6 | fetch failed |
| offset:13 | ❌ ERROR | 10.6 | fetch failed |
| offset:14 | ❌ ERROR | 10.6 | fetch failed |
| offset:15 | ❌ ERROR | 10.6 | fetch failed |
| offset:16 | ❌ ERROR | 10.6 | fetch failed |
| offset:17 | ❌ ERROR | 10.6 | fetch failed |
| offset:18 | ❌ ERROR | 10.6 | fetch failed |
| offset:19 | ❌ ERROR | 10.6 | fetch failed |
| offset:20 | ❌ ERROR | 10.6 | fetch failed |
| offset:21 | ❌ ERROR | 10.6 | fetch failed |
| offset:22 | ❌ ERROR | 10.6 | fetch failed |
| offset:23 | ❌ ERROR | 10.6 | fetch failed |
| offset:24 | ❌ ERROR | 10.6 | fetch failed |
| offset:25 | ❌ ERROR | 10.6 | fetch failed |
| offset:26 | ❌ ERROR | 10.6 | fetch failed |
| offset:27 | ❌ ERROR | 10.6 | fetch failed |
| offset:28 | ❌ ERROR | 10.6 | fetch failed |
| offset:29 | ❌ ERROR | 10.6 | fetch failed |
| offset:30 | ❌ ERROR | 10.6 | fetch failed |
| offset:31 | ❌ ERROR | 10.6 | fetch failed |
| offset:32 | ❌ ERROR | 10.6 | fetch failed |
| offset:33 | ❌ ERROR | 10.6 | fetch failed |
| offset:34 | ❌ ERROR | 10.6 | fetch failed |
| offset:35 | ❌ ERROR | 10.6 | fetch failed |
| offset:36 | ❌ ERROR | 10.6 | fetch failed |
| offset:37 | ❌ ERROR | 10.6 | fetch failed |
| offset:38 | ❌ ERROR | 10.6 | fetch failed |
| offset:39 | ❌ ERROR | 10.6 | fetch failed |
| offset:40 | ❌ ERROR | 10.6 | fetch failed |
| offset:41 | ❌ ERROR | 10.6 | fetch failed |
| offset:42 | ❌ ERROR | 10.6 | fetch failed |
| offset:43 | ❌ ERROR | 10.6 | fetch failed |
| offset:44 | ❌ ERROR | 10.6 | fetch failed |
| offset:45 | ❌ ERROR | 10.6 | fetch failed |
| offset:46 | ❌ ERROR | 10.6 | fetch failed |
| offset:47 | ❌ ERROR | 10.6 | fetch failed |
| offset:48 | ❌ ERROR | 10.6 | fetch failed |
| offset:49 | ❌ ERROR | 10.6 | fetch failed |
| Security: FortiOS Impact Analysis | ❌ FAIL | 84.9 | Missing [DEBRIEF_V1_JSON] block |
| offset:51 | ❌ ERROR | 10.6 | fetch failed |
| offset:52 | ❌ ERROR | 10.6 | fetch failed |
| offset:53 | ❌ ERROR | 10.6 | fetch failed |
| offset:54 | ❌ ERROR | 10.6 | fetch failed |
| offset:55 | ❌ ERROR | 10.6 | fetch failed |
| offset:56 | ❌ ERROR | 10.6 | fetch failed |
| offset:57 | ❌ ERROR | 10.6 | fetch failed |
| offset:58 | ❌ ERROR | 10.6 | fetch failed |
| offset:59 | ❌ ERROR | 10.6 | fetch failed |
| offset:60 | ❌ ERROR | 10.6 | fetch failed |
| offset:61 | ❌ ERROR | 10.6 | fetch failed |
| offset:62 | ❌ ERROR | 10.6 | fetch failed |
| offset:63 | ❌ ERROR | 10.6 | fetch failed |
| offset:64 | ❌ ERROR | 10.6 | fetch failed |
| offset:65 | ❌ ERROR | 10.6 | fetch failed |
| offset:66 | ❌ ERROR | 10.6 | fetch failed |
| offset:67 | ❌ ERROR | 10.6 | fetch failed |
| offset:68 | ❌ ERROR | 10.6 | fetch failed |
| offset:69 | ❌ ERROR | 10.6 | fetch failed |
| offset:70 | ❌ ERROR | 10.6 | fetch failed |
| offset:71 | ❌ ERROR | 10.6 | fetch failed |
| offset:72 | ❌ ERROR | 10.6 | fetch failed |
| offset:73 | ❌ ERROR | 10.6 | fetch failed |
| offset:74 | ❌ ERROR | 10.6 | fetch failed |
| offset:75 | ❌ ERROR | 10.6 | fetch failed |
| offset:76 | ❌ ERROR | 10.6 | fetch failed |
| offset:77 | ❌ ERROR | 10.6 | fetch failed |
| offset:78 | ❌ ERROR | 10.6 | fetch failed |
| offset:79 | ❌ ERROR | 10.6 | fetch failed |
| offset:80 | ❌ ERROR | 10.6 | fetch failed |
| offset:81 | ❌ ERROR | 10.6 | fetch failed |
| offset:82 | ❌ ERROR | 10.6 | fetch failed |
| offset:83 | ❌ ERROR | 10.6 | fetch failed |
| offset:84 | ❌ ERROR | 10.6 | fetch failed |
| offset:85 | ❌ ERROR | 10.6 | fetch failed |
| offset:86 | ❌ ERROR | 10.6 | fetch failed |
| offset:87 | ❌ ERROR | 10.6 | fetch failed |
| offset:88 | ❌ ERROR | 10.6 | fetch failed |
| offset:89 | ❌ ERROR | 10.6 | fetch failed |
| offset:90 | ❌ ERROR | 10.6 | fetch failed |
| offset:91 | ❌ ERROR | 10.6 | fetch failed |
| offset:92 | ❌ ERROR | 10.6 | fetch failed |
| offset:93 | ❌ ERROR | 10.6 | fetch failed |
| offset:94 | ❌ ERROR | 10.6 | fetch failed |
| offset:95 | ❌ ERROR | 10.6 | fetch failed |
| offset:96 | ❌ ERROR | 10.6 | fetch failed |
| offset:97 | ❌ ERROR | 10.6 | fetch failed |
| offset:98 | ❌ ERROR | 10.6 | fetch failed |
| offset:99 | ❌ ERROR | 10.6 | fetch failed |

## Disclosure Metrics (P0 Instrumentation)

### Disclosure Level by Model

| Model | Full | Partial | None |
|-------|------|---------|------|
| claude-haiku-4.5 | 0 | 1 | 93 |
| gpt-5-mini | 0 | 0 | 93 |
| gemini-3-flash | 0 | 0 | 88 |
| deepseek-r1 | 0 | 0 | 16 |
| deepseek-v3.2 | 0 | 0 | 0 |
| qwen3-235b | 0 | 0 | 0 |
| minimax-m2.1 | 0 | 0 | 0 |
| mimo-v2-flash-free | 0 | 0 | 2 |

### Disclosure Warnings (Non-Scored)

> **Note:** These warnings are informational and do not affect pass/fail.
> They track progressive disclosure usage for Week 1-2 observability.

⚠️ [claude-haiku-4.5/fin_revenue_apple_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_revenue_apple_2024] No skill search for ENTERPRISE_EXEC scenario
⚠️ [claude-haiku-4.5/fin_revenue_apple_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_revenue_google_2024] No skill search for EARLY_STAGE_VC scenario
⚠️ [claude-haiku-4.5/fin_revenue_nvidia_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_revenue_nvidia_2024] No skill search for FOUNDER_STRATEGY scenario
⚠️ [claude-haiku-4.5/fin_revenue_nvidia_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_revenue_amazon_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_revenue_amazon_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_revenue_meta_2024] No skill search for ENTERPRISE_EXEC scenario
⚠️ [claude-haiku-4.5/fin_revenue_amd_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_revenue_amd_2024] No skill search for QUANT_ANALYST scenario
⚠️ [claude-haiku-4.5/fin_revenue_amd_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_revenue_intel_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_revenue_intel_2024] No skill search for CTO_TECH_LEAD scenario
⚠️ [claude-haiku-4.5/fin_revenue_intel_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_revenue_qualcomm_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_revenue_qualcomm_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_risk_google_2024] No skill search for ENTERPRISE_EXEC scenario
⚠️ [claude-haiku-4.5/fin_risk_nvidia_2024] No skill search for CTO_TECH_LEAD scenario
⚠️ [claude-haiku-4.5/fin_risk_apple_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_risk_apple_2024] No skill search for ENTERPRISE_EXEC scenario
⚠️ [claude-haiku-4.5/fin_risk_apple_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_risk_microsoft_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_risk_microsoft_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_risk_tesla_2024] No skill search for EARLY_STAGE_VC scenario
⚠️ [claude-haiku-4.5/fin_risk_amazon_2024] No skill search for ECOSYSTEM_PARTNER scenario
⚠️ [claude-haiku-4.5/fin_risk_meta_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_risk_meta_2024] No skill search for FOUNDER_STRATEGY scenario
⚠️ [claude-haiku-4.5/fin_risk_meta_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_risk_amd_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_risk_amd_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_risk_intel_2024] No skill search for QUANT_ANALYST scenario
⚠️ [claude-haiku-4.5/fin_risk_qualcomm_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_risk_qualcomm_2024] No skill search for SALES_ENGINEER scenario
⚠️ [claude-haiku-4.5/fin_risk_qualcomm_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_business_amd_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_business_amd_2024] No skill search for PRODUCT_DESIGNER scenario
⚠️ [claude-haiku-4.5/fin_business_amd_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_business_nvidia_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_business_nvidia_2024] No skill search for FOUNDER_STRATEGY scenario
⚠️ [claude-haiku-4.5/fin_business_nvidia_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_business_cloud_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_business_cloud_2024] No skill search for EARLY_STAGE_VC scenario
⚠️ [claude-haiku-4.5/fin_business_cloud_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_business_semiconductor_2024] No skill search for CTO_TECH_LEAD scenario
⚠️ [claude-haiku-4.5/fin_business_social_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_business_social_2024] No progressive disclosure meta-tools used
⚠️ [claude-haiku-4.5/fin_compare_cloud_2024] No skill search before tool invoke
⚠️ [claude-haiku-4.5/fin_compare_cloud_2024] No skill search for EARLY_STAGE_VC scenario
... and 433 more warnings
