# Multi-SDK Benchmark Report

**Date:** 2026-03-22T01:38:32.003Z
**Duration:** 963.6s
**Runs:** 8 (6 success, 2 failed)
**Success Rate:** 75.0%

## SDK Comparison

| SDK    | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ------ | -------- | --------- | -------- | -------- | ------- |
| OpenAI | balanced | 77.2      | 1.3m     | $1.0631  | 75%     |

## Variant Rankings (by avg score)

| Rank | Variant              | Score | Time | Cost    | Success | Tier Eval |
| ---- | -------------------- | ----- | ---- | ------- | ------- | --------- |
| 1    | openai-multi-agent-0 | 77.2  | 1.3m | $1.0631 | 75%     | n/a       |

## Per-Commit Results

### Weather Card (simple)

- **Avg Score:** 79.0
- **Best:** openai-multi-agent-0
- **Worst:** openai-multi-agent-0

### Multi-Step Survey Form (medium)

- **Avg Score:** 78.0
- **Best:** openai-multi-agent-0
- **Worst:** openai-multi-agent-0

### Kanban Task Board (medium)

- **Avg Score:** 0.0
- **Best:** none
- **Worst:** none

### Interactive Periodic Table (complex)

- **Avg Score:** 71.6
- **Best:** openai-multi-agent-0
- **Worst:** openai-multi-agent-0

### E-Commerce Product Page (complex)

- **Avg Score:** 80.6
- **Best:** openai-multi-agent-0
- **Worst:** openai-multi-agent-0

### Chat Interface (medium)

- **Avg Score:** 75.0
- **Best:** openai-multi-agent-0
- **Worst:** openai-multi-agent-0

### Stock Ticker Dashboard (simple)

- **Avg Score:** 79.0
- **Best:** openai-multi-agent-0
- **Worst:** openai-multi-agent-0

### Onboarding Wizard (medium)

- **Avg Score:** 0.0
- **Best:** none
- **Worst:** none

## Failures

- **openai-multi-agent-0** × kanban-board: Enforced coding loop: all 10 attempts failed (886424 tokens)
  Last error: ## compile_component errors
  {
  "success": false,
  "phase": "validation",
  "error": "## Validation Errors (must fix)\n\n- **SIZE**: Component has too many lines: 573 exceeds 500 line limit\n - Fix: Split the component into smaller sub-components.\n\n## General Suggestions\n\n- Fix all errors before compiling - the component will not work with errors.\n\n## Stats: 573 lines, 15 primitives used"
  }
- **openai-multi-agent-0** × onboarding-wizard: Timeout after 300000ms
