# Multi-SDK Benchmark Report

**Date:** 2026-03-22T16:59:58.024Z
**Duration:** 300.1s
**Runs:** 16 (15 success, 1 failed)
**Success Rate:** 93.8%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| Anthropic Claude | balanced | 81.6      | 33.0s    | $0.0963  | 100%    |
| OpenAI           | balanced | 67.4      | 44.6s    | $0.4257  | 88%     |

## Variant Rankings (by avg score)

| Rank | Variant          | Score | Time  | Cost    | Success | Tier Eval |
| ---- | ---------------- | ----- | ----- | ------- | ------- | --------- |
| 1    | claude-default-0 | 81.6  | 33.0s | $0.0963 | 100%    | 1P/4W/2F  |
| 2    | openai-default-1 | 67.4  | 44.6s | $0.4257 | 88%     | 2P/4W/1F  |

## Per-Commit Results

### Weather Card (simple)

- **Avg Score:** 82.0
- **Best:** claude-default-0
- **Worst:** openai-default-1
- **Tier Eval:** 0 pass, 1 warn, 1 fail

### Multi-Step Survey Form (medium)

- **Avg Score:** 84.0
- **Best:** claude-default-0
- **Worst:** openai-default-1
- **Tier Eval:** 2 pass, 0 warn, 0 fail

### Kanban Task Board (medium)

- **Avg Score:** 56.0
- **Best:** claude-default-0
- **Worst:** openai-default-1
- **Tier Eval:** 0 pass, 2 warn, 0 fail

### Interactive Periodic Table (complex)

- **Avg Score:** 66.0
- **Best:** claude-default-0
- **Worst:** openai-default-1
- **Tier Eval:** 0 pass, 2 warn, 0 fail

### E-Commerce Product Page (complex)

- **Avg Score:** 70.5
- **Best:** claude-default-0
- **Worst:** openai-default-1
- **Tier Eval:** 0 pass, 0 warn, 1 fail

### Chat Interface (medium)

- **Avg Score:** 79.0
- **Best:** claude-default-0
- **Worst:** claude-default-0
- **Tier Eval:** 0 pass, 1 warn, 0 fail

### Stock Ticker Dashboard (simple)

- **Avg Score:** 78.7
- **Best:** openai-default-1
- **Worst:** claude-default-0
- **Tier Eval:** 0 pass, 1 warn, 1 fail

### Onboarding Wizard (medium)

- **Avg Score:** 85.7
- **Best:** openai-default-1
- **Worst:** claude-default-0
- **Tier Eval:** 1 pass, 1 warn, 0 fail

## Failures

- **openai-default-1** × chat-interface: Timeout after 300000ms
