# Multi-SDK Benchmark Report

**Date:** 2026-03-18T02:45:53.731Z
**Duration:** 600.0s
**Runs:** 36 (33 success, 3 failed)
**Success Rate:** 91.7%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| Anthropic Claude | balanced | 74.7      | 2.3m     | $0.5107  | 92%     |
| OpenAI           | balanced | 82.6      | 28.3s    | $0.0485  | 100%    |
| Google Gemini    | balanced | 75.9      | 3.0m     | $0.5627  | 83%     |

## Variant Rankings (by avg score)

| Rank | Variant      | Score | Time  | Cost    | Success |
| ---- | ------------ | ----- | ----- | ------- | ------- |
| 1    | openai-raw-2 | 85.9  | 27.0s | $0.0970 | 100%    |
| 2    | openai-sdk-3 | 79.3  | 29.7s | $0.0000 | 100%    |
| 3    | google-sdk-5 | 77.4  | 2.9m  | $0.4699 | 83%     |
| 4    | claude-raw-0 | 74.7  | 1.9m  | $0.6938 | 100%    |
| 5    | google-raw-4 | 74.5  | 3.2m  | $0.6556 | 83%     |
| 6    | claude-sdk-1 | n/a   | 2.8m  | $0.2911 | 83%     |

## Per-Commit Results

### Weather Card (simple)

- **Avg Score:** 81.5
- **Best:** openai-sdk-3
- **Worst:** google-sdk-5

### Kanban Task Board (medium)

- **Avg Score:** 64.8
- **Best:** openai-raw-2
- **Worst:** openai-sdk-3

### Interactive Periodic Table (complex)

- **Avg Score:** 79.2
- **Best:** openai-sdk-3
- **Worst:** google-raw-4

### Chat Interface (medium)

- **Avg Score:** 81.6
- **Best:** openai-raw-2
- **Worst:** google-raw-4

### Stock Ticker Dashboard (simple)

- **Avg Score:** 79.6
- **Best:** openai-raw-2
- **Worst:** claude-raw-0

### Onboarding Wizard (medium)

- **Avg Score:** 79.0
- **Best:** openai-raw-2
- **Worst:** google-sdk-5

## Failures

- **claude-sdk-1** × onboarding-wizard: Claude SDK iterative: no compiled code produced
- **google-raw-4** × kanban-board: Timeout after 600000ms
- **google-sdk-5** × kanban-board: Timeout after 600000ms
