# Expanded Evaluation Results

Generated: 2026-04-21T17:30:53.462Z
Model: gpt-5-mini
Category: all
Total Time: 2092.9s

## Summary

| Metric | Value |
|--------|-------|
| Total Scenarios | 31 |
| Passed | 19 |
| Failed | 12 |
| Pass Rate | 61.3% |

## Results by Category

### Calendar

| Scenario | Status | Time | Tools Called | Issues |
|----------|--------|------|--------------|--------|
| Calendar: Schedule DISCO review meeting | ✅ PASS | 54.6s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity, askHuman | - |
| Calendar: Schedule Salesforce demo | ✅ PASS | 42.7s | searchAvailableSkills, lookupGroundTruthEntity, linkupSearch, searchAvailableSkills, lookupGroundTruthEntity | - |
| Calendar: Schedule Ambros follow-up | ✅ PASS | 44.0s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | - |

### Spreadsheet

| Scenario | Status | Time | Tools Called | Issues |
|----------|--------|------|--------------|--------|
| Spreadsheet: DISCO funding metrics | ✅ PASS | 41.1s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity, createSpreadsheet | - |
| Spreadsheet: Ambros deal pipeline | ✅ PASS | 53.8s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity, listSpreadsheets, createSpreadsheet | - |
| Spreadsheet: Gemini pricing comparison | ✅ PASS | 40.8s | searchAvailableSkills, lookupGroundTruthEntity, linkupSearch, searchAvailableSkills, lookupGroundTruthEntity, createSpreadsheet | - |

### Document

| Scenario | Status | Time | Tools Called | Issues |
|----------|--------|------|--------------|--------|
| Document: DISCO investment memo | ✅ PASS | 42.5s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | - |
| Document: QuickJS security assessment | ✅ PASS | 71.9s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity, linkupSearch | - |
| Document: GenomiQ due diligence | ❌ FAIL | 64.9s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | persona mismatch: got JPM_STARTUP_BANKER expected EARLY_STAG |

### Web

| Scenario | Status | Time | Tools Called | Issues |
|----------|--------|------|--------------|--------|
| Web: DISCO news search | ❌ FAIL | 127.0s | none | Missing required tool call: lookupGroundTruthEntity; Output  |
| Web: Salesforce Agentforce deep dive | ❌ FAIL | 135.2s | none | Missing required tool call: lookupGroundTruthEntity; Output  |
| Web: DISCO multi-source verification | ❌ FAIL | 145.4s | none | Missing required tool call: lookupGroundTruthEntity; Output  |

### Media

| Scenario | Status | Time | Tools Called | Issues |
|----------|--------|------|--------------|--------|
| Media: DISCO UI card design | ✅ PASS | 55.4s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | - |
| Media: SoundCloud incident timeline | ❌ FAIL | 67.2s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | persona mismatch: got JPM_STARTUP_BANKER expected CTO_TECH_L |
| Media: VaultPay sales materials | ❌ FAIL | 133.4s | none | Missing required tool call: lookupGroundTruthEntity; Output  |

### Skills

| Scenario | Status | Time | Tools Called | Issues |
|----------|--------|------|--------------|--------|
| Skills: Banker skill discovery | ✅ PASS | 54.1s | searchAvailableSkills, lookupGroundTruthEntity, initScratchpad, searchAvailableSkills, lookupGroundTruthEntity | - |
| Skills: VC thesis skill | ❌ FAIL | 66.9s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | unknown ground truth entity: OPENAUTOGLM |
| Skills: Quant signal skill | ✅ PASS | 83.6s | searchAvailableSkills, lookupGroundTruthEntity, initScratchpad, searchAvailableSkills, lookupGroundTruthEntity, describeSkill | - |
| Skills: Digest persona skill | ❌ FAIL | 76.2s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, getDailyBrief, getLiveFeed | hqLocation does not match ground truth; funding.stage mismat |

### Tools

| Scenario | Status | Time | Tools Called | Issues |
|----------|--------|------|--------------|--------|
| Tools: Basic tool discovery | ✅ PASS | 60.5s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | - |
| Tools: Describe tool schemas | ❌ FAIL | 74.6s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | Output missing expected content: "schema"; Output missing ex |
| Tools: Gateway invocation | ✅ PASS | 49.4s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | - |
| Tools: Safe tier tool | ✅ PASS | 67.9s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | - |

### Hybrid

| Scenario | Status | Time | Tools Called | Issues |
|----------|--------|------|--------------|--------|
| Hybrid: Memory-first retrieval | ✅ PASS | 67.4s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | - |
| Hybrid: Fusion search ranking | ✅ PASS | 55.6s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | - |
| Hybrid: Cross-source aggregation | ❌ FAIL | 75.2s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity, queryMemory, linkupSearch | persona mismatch: got JPM_STARTUP_BANKER expected EARLY_STAG |
| Hybrid: Stale data detection | ❌ FAIL | 39.7s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | funding.stage mismatch: got 'N/A' expected 'Series A (report |

### Disclosure

| Scenario | Status | Time | Tools Called | Issues |
|----------|--------|------|--------------|--------|
| Disclosure: P0 telemetry events | ❌ FAIL | 52.2s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | persona mismatch: got QUANT_ANALYST expected JPM_STARTUP_BAN |
| Disclosure: P1 persona classification | ✅ PASS | 43.6s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | - |
| Disclosure: P2 skill allowlist | ✅ PASS | 55.9s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | - |
| Disclosure: P3 full disclosure trace | ✅ PASS | 50.2s | searchAvailableSkills, lookupGroundTruthEntity, searchAvailableSkills, lookupGroundTruthEntity | - |
