# Multi-SDK Benchmark Report

**Date:** 2026-03-18T07:08:17.420Z
**Duration:** 900.1s
**Runs:** 72 (64 success, 8 failed)
**Success Rate:** 88.9%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| OpenAI           | balanced | 81.6      | 22.4s    | $0.0629  | 96%     |
| Google Gemini    | balanced | 78.4      | 2.6m     | $0.3788  | 96%     |
| Anthropic Claude | balanced | 80.3      | 3.2m     | $0.6260  | 75%     |

## Variant Rankings (by avg score)

| Rank | Variant             | Score | Time  | Cost    | Success |
| ---- | ------------------- | ----- | ----- | ------- | ------- |
| 1    | claude-raw-c4835-10 | 82.3  | 4.8m  | $0.5776 | 67%     |
| 2    | openai-raw-c1-0     | 82.1  | 22.7s | $0.1328 | 83%     |
| 3    | openai-sdk-c0437-3  | 82.0  | 19.7s | $0.0000 | 100%    |
| 4    | claude-sdk-c1-9     | 82.0  | 2.0m  | $0.2405 | 67%     |
| 5    | openai-sdk-c1-1     | 82.0  | 24.5s | $0.0000 | 100%    |
| 6    | openai-raw-c0437-2  | 80.5  | 22.8s | $0.1304 | 100%    |
| 7    | google-raw-c3975-6  | 80.1  | 2.3m  | $0.3489 | 100%    |
| 8    | google-sdk-c3975-7  | 78.7  | 2.3m  | $0.3614 | 100%    |
| 9    | claude-raw-c1-8     | 78.7  | 2.3m  | $0.8637 | 100%    |
| 10   | google-raw-c1-4     | 77.7  | 2.5m  | $0.3671 | 100%    |
| 11   | google-sdk-c1-5     | 76.8  | 3.2m  | $0.4497 | 83%     |
| 12   | claude-sdk-c4835-11 | n/a   | 4.1m  | $0.7033 | 67%     |

## Per-Commit Results

### Weather Card (simple)

- **Avg Score:** 80.7
- **Best:** google-raw-c3975-6
- **Worst:** google-sdk-c1-5

### Multi-Step Survey Form (medium)

- **Avg Score:** 78.8
- **Best:** openai-sdk-c1-1
- **Worst:** google-sdk-c1-5

### Interactive Periodic Table (complex)

- **Avg Score:** 77.5
- **Best:** openai-raw-c0437-2
- **Worst:** google-raw-c1-4

### Chat Interface (medium)

- **Avg Score:** 78.7
- **Best:** claude-raw-c4835-10
- **Worst:** openai-sdk-c0437-3

### Stock Ticker Dashboard (simple)

- **Avg Score:** 83.8
- **Best:** openai-raw-c1-0
- **Worst:** claude-raw-c1-8

### Onboarding Wizard (medium)

- **Avg Score:** 80.7
- **Best:** openai-raw-c0437-2
- **Worst:** google-sdk-c3975-7

## Failures

- **openai-raw-c1-0** × periodic-table: OpenAI raw iterative: no compiled code produced (1 turns, 42176 tokens)
  No tool calls made — model responded with text only
- **google-sdk-c1-5** × onboarding-wizard: API key must be provided via constructor or GOOGLE_GENAI_API_KEY or GEMINI_API_KEY environment variable.
- **claude-sdk-c1-9** × survey-form: Claude SDK iterative: no compiled code produced (246.9s elapsed, 31 turns)
  sourceCode captured: false, meta captured: false
- **claude-sdk-c1-9** × onboarding-wizard: Claude SDK iterative: no compiled code produced (161.1s elapsed, 23 turns)
  sourceCode captured: false, meta captured: false
- **claude-raw-c4835-10** × periodic-table: Timeout after 900000ms
- **claude-raw-c4835-10** × stock-ticker: Timeout after 900000ms
- **claude-sdk-c4835-11** × chat-interface: Timeout after 900000ms
- **claude-sdk-c4835-11** × onboarding-wizard: Timeout after 900000ms
