# Multi-SDK Benchmark Report

**Date:** 2026-03-18T03:47:17.650Z
**Duration:** 989.3s
**Runs:** 72 (68 success, 4 failed)
**Success Rate:** 94.4%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| Anthropic Claude | balanced | 79.2      | 2.7m     | $0.7010  | 88%     |
| OpenAI           | balanced | 78.6      | 1.1m     | $0.1262  | 100%    |
| Google Gemini    | balanced | 78.5      | 3.9m     | $0.6293  | 96%     |

## Variant Rankings (by avg score)

| Rank | Variant             | Score | Time  | Cost    | Success |
| ---- | ------------------- | ----- | ----- | ------- | ------- |
| 1    | claude-sdk-c1-1     | 88.0  | 1.2m  | $0.1644 | 50%     |
| 2    | openai-sdk-c1-5     | 80.6  | 27.8s | $0.0000 | 100%    |
| 3    | google-sdk-c1-9     | 79.8  | 4.1m  | $1.1268 | 100%    |
| 4    | google-sdk-c2166-11 | 79.2  | 3.3m  | $0.2920 | 83%     |
| 5    | openai-raw-c1-4     | 79.2  | 26.0s | $0.1159 | 100%    |
| 6    | claude-raw-c1-0     | 78.6  | 2.6m  | $0.9495 | 100%    |
| 7    | claude-raw-c7532-2  | 78.4  | 3.2m  | $0.4755 | 100%    |
| 8    | openai-sdk-c0682-7  | 78.2  | 1.9m  | $0.0000 | 100%    |
| 9    | google-raw-c1-8     | 77.5  | 2.8m  | $0.5087 | 100%    |
| 10   | google-raw-c2166-10 | 77.4  | 5.2m  | $0.5334 | 100%    |
| 11   | openai-raw-c0682-6  | 76.4  | 1.6m  | $0.3891 | 100%    |
| 12   | claude-sdk-c7532-3  | n/a   | 3.0m  | $0.9465 | 100%    |

## Per-Commit Results

### Weather Card (simple)

- **Avg Score:** 80.9
- **Best:** claude-sdk-c1-1
- **Worst:** claude-raw-c7532-2

### Kanban Task Board (medium)

- **Avg Score:** 70.5
- **Best:** openai-sdk-c1-5
- **Worst:** claude-raw-c1-0

### Interactive Periodic Table (complex)

- **Avg Score:** 79.1
- **Best:** google-raw-c1-8
- **Worst:** openai-raw-c0682-6

### Chat Interface (medium)

- **Avg Score:** 81.3
- **Best:** claude-raw-c1-0
- **Worst:** google-sdk-c1-9

### Stock Ticker Dashboard (simple)

- **Avg Score:** 79.3
- **Best:** openai-raw-c1-4
- **Worst:** openai-raw-c0682-6

### Onboarding Wizard (medium)

- **Avg Score:** 80.1
- **Best:** google-sdk-c2166-11
- **Worst:** google-raw-c1-8

## Failures

- **claude-sdk-c1-1** × kanban-board: Claude SDK iterative: no compiled code produced
- **claude-sdk-c1-1** × chat-interface: Claude SDK iterative: no compiled code produced
- **claude-sdk-c1-1** × onboarding-wizard: Claude SDK iterative: no compiled code produced
- **google-sdk-c2166-11** × kanban-board: Google ADK iterative: no compiled code produced
