# Multi-SDK Benchmark Report

**Date:** 2026-03-18T03:17:41.640Z
**Duration:** 600.1s
**Runs:** 72 (63 success, 9 failed)
**Success Rate:** 87.5%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| OpenAI           | balanced | 83.9      | 23.8s    | $0.0592  | 96%     |
| Google Gemini    | balanced | 77.2      | 2.2m     | $0.3687  | 92%     |
| Anthropic Claude | balanced | 79.9      | 3.5m     | $0.6302  | 75%     |

## Variant Rankings (by avg score)

| Rank | Variant             | Score | Time  | Cost    | Success |
| ---- | ------------------- | ----- | ----- | ------- | ------- |
| 1    | openai-raw-c1-0     | 84.6  | 25.3s | $0.1274 | 100%    |
| 2    | openai-sdk-c0437-3  | 84.5  | 21.4s | $0.0000 | 100%    |
| 3    | openai-sdk-c1-1     | 83.6  | 24.3s | $0.0000 | 83%     |
| 4    | openai-raw-c0437-2  | 82.7  | 24.1s | $0.0997 | 100%    |
| 5    | claude-raw-c4835-10 | 80.3  | 5.3m  | $0.5566 | 67%     |
| 6    | claude-raw-c1-8     | 79.6  | 2.5m  | $0.9394 | 83%     |
| 7    | google-sdk-c1-5     | 78.6  | 1.8m  | $0.2583 | 83%     |
| 8    | google-sdk-c3975-7  | 77.9  | 2.2m  | $0.3775 | 100%    |
| 9    | google-raw-c3975-6  | 76.8  | 2.2m  | $0.3763 | 83%     |
| 10   | google-raw-c1-4     | 75.7  | 2.6m  | $0.4455 | 100%    |
| 11   | claude-sdk-c1-9     | n/a   | 2.2m  | $0.2497 | 67%     |
| 12   | claude-sdk-c4835-11 | n/a   | 4.2m  | $0.6844 | 83%     |

## Per-Commit Results

### Weather Card (simple)

- **Avg Score:** 82.4
- **Best:** openai-raw-c1-0
- **Worst:** google-sdk-c3975-7

### Multi-Step Survey Form (medium)

- **Avg Score:** 78.4
- **Best:** google-sdk-c1-5
- **Worst:** google-raw-c1-4

### Interactive Periodic Table (complex)

- **Avg Score:** 79.5
- **Best:** claude-raw-c1-8
- **Worst:** google-raw-c1-4

### Chat Interface (medium)

- **Avg Score:** 79.1
- **Best:** openai-raw-c0437-2
- **Worst:** google-raw-c1-4

### Stock Ticker Dashboard (simple)

- **Avg Score:** 82.8
- **Best:** openai-sdk-c1-1
- **Worst:** google-raw-c3975-6

### Onboarding Wizard (medium)

- **Avg Score:** 80.2
- **Best:** openai-sdk-c0437-3
- **Worst:** google-sdk-c1-5

## Failures

- **openai-sdk-c1-1** × chat-interface: OpenAI SDK iterative: no compiled code produced
- **google-sdk-c1-5** × chat-interface: API key must be provided via constructor or GOOGLE_GENAI_API_KEY or GEMINI_API_KEY environment variable.
- **google-raw-c3975-6** × survey-form: Timeout after 600000ms
- **claude-raw-c1-8** × onboarding-wizard: Claude raw iterative: no compiled code produced
- **claude-sdk-c1-9** × survey-form: Claude SDK iterative: no compiled code produced
- **claude-sdk-c1-9** × onboarding-wizard: Claude SDK iterative: no compiled code produced
- **claude-raw-c4835-10** × periodic-table: Timeout after 600000ms
- **claude-raw-c4835-10** × chat-interface: Timeout after 600000ms
- **claude-sdk-c4835-11** × chat-interface: Timeout after 600000ms
