# Multi-SDK Benchmark Report

**Date:** 2026-03-17T11:02:57.246Z
**Duration:** 600.0s
**Runs:** 48 (28 success, 20 failed)
**Success Rate:** 58.3%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| Anthropic Claude | balanced | 81.2      | 2.7m     | $0.3101  | 63%     |
| Google Gemini    | balanced | 77.5      | 2.5m     | $0.2068  | 92%     |
| OpenAI           | balanced | 87.8      | 18.9s    | $0.0574  | 17%     |

## Variant Rankings (by avg score)

| Rank | Variant            | Score | Time  | Cost    | Success |
| ---- | ------------------ | ----- | ----- | ------- | ------- |
| 1    | openai-raw-6       | 88.0  | 20.2s | $0.1149 | 17%     |
| 2    | openai-sdk-7       | 87.6  | 17.6s | $0.0000 | 17%     |
| 3    | claude-raw-c1-0    | 81.5  | 1.3m  | $0.3119 | 33%     |
| 4    | claude-raw-c4835-2 | 81.0  | 3.7m  | $0.4203 | 83%     |
| 5    | google-sdk-5       | 78.3  | 2.4m  | $0.1773 | 100%    |
| 6    | google-raw-4       | 76.5  | 2.7m  | $0.2423 | 83%     |
| 7    | claude-sdk-c1-1    | n/a   | 2.3m  | $0.1702 | 83%     |
| 8    | claude-sdk-c4835-3 | n/a   | 2.4m  | $0.3581 | 50%     |

## Per-Commit Results

### Weather Card (simple)

- **Avg Score:** 81.3
- **Best:** claude-raw-c4835-2
- **Worst:** claude-raw-c1-0

### Multi-Step Survey Form (medium)

- **Avg Score:** 78.9
- **Best:** google-raw-4
- **Worst:** claude-raw-c4835-2

### Interactive Periodic Table (complex)

- **Avg Score:** 75.8
- **Best:** google-sdk-5
- **Worst:** google-raw-4

### Chat Interface (medium)

- **Avg Score:** 75.5
- **Best:** claude-raw-c4835-2
- **Worst:** google-raw-4

### Stock Ticker Dashboard (simple)

- **Avg Score:** 84.6
- **Best:** claude-raw-c1-0
- **Worst:** claude-raw-c4835-2

### Onboarding Wizard (medium)

- **Avg Score:** 77.5
- **Best:** claude-raw-c4835-2
- **Worst:** google-sdk-5

## Failures

- **claude-raw-c1-0** × survey-form: Claude raw iterative: no compiled code produced
- **claude-raw-c1-0** × periodic-table: Claude raw iterative: no compiled code produced
- **claude-raw-c1-0** × chat-interface: Claude raw iterative: no compiled code produced
- **claude-raw-c1-0** × onboarding-wizard: Claude raw iterative: no compiled code produced
- **claude-sdk-c1-1** × periodic-table: Claude SDK iterative: no compiled code produced
- **claude-raw-c4835-2** × periodic-table: Timeout after 600000ms
- **claude-sdk-c4835-3** × periodic-table: Timeout after 600000ms
- **claude-sdk-c4835-3** × chat-interface: Timeout after 600000ms
- **claude-sdk-c4835-3** × onboarding-wizard: Timeout after 600000ms
- **google-raw-4** × onboarding-wizard: Google raw iterative: no compiled code produced
- **openai-raw-6** × survey-form: OpenAI raw iterative: no compiled code produced
- **openai-raw-6** × periodic-table: OpenAI raw iterative: no compiled code produced
- **openai-raw-6** × chat-interface: OpenAI raw iterative: no compiled code produced
- **openai-raw-6** × stock-ticker: OpenAI raw iterative: no compiled code produced
- **openai-raw-6** × onboarding-wizard: OpenAI raw iterative: no compiled code produced
- **openai-sdk-7** × survey-form: OpenAI SDK iterative: no compiled code produced
- **openai-sdk-7** × periodic-table: OpenAI SDK iterative: no compiled code produced
- **openai-sdk-7** × chat-interface: OpenAI SDK iterative: no compiled code produced
- **openai-sdk-7** × stock-ticker: OpenAI SDK iterative: no compiled code produced
- **openai-sdk-7** × onboarding-wizard: OpenAI SDK iterative: no compiled code produced
