# Multi-SDK Benchmark Report

**Date:** 2026-03-17T07:41:50.592Z
**Duration:** 600.0s
**Runs:** 40 (18 success, 22 failed)
**Success Rate:** 45.0%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| Anthropic Claude | balanced | n/a       | 3.3m     | $0.2617  | 45%     |
| Google Gemini    | balanced | n/a       | 1.8m     | $0.1007  | 80%     |
| OpenAI           | balanced | n/a       | 58.5s    | $0.0956  | 10%     |

## Variant Rankings (by avg score)

| Rank | Variant                      | Score | Time  | Cost    | Success |
| ---- | ---------------------------- | ----- | ----- | ------- | ------- |
| 1    | openai-sdk-iterative-7       | 0.0   | 0ms   | $0.0000 | 0%      |
| 2    | claude-raw-iterative-c1-0    | n/a   | 55.5s | $0.1038 | 20%     |
| 3    | claude-sdk-iterative-c1-1    | n/a   | 2.4m  | $0.2086 | 60%     |
| 4    | claude-raw-iterative-c4835-2 | n/a   | 4.1m  | $0.2520 | 80%     |
| 5    | claude-sdk-iterative-c4835-3 | n/a   | 5.2m  | $0.6176 | 20%     |
| 6    | google-raw-iterative-4       | n/a   | 2.0m  | $0.1155 | 80%     |
| 7    | google-sdk-iterative-5       | n/a   | 1.6m  | $0.0859 | 80%     |
| 8    | openai-raw-iterative-6       | n/a   | 58.5s | $0.0956 | 20%     |

## Per-Prompt Results

### Weather Card (simple)

- **Avg Score:** -1.0
- **Best:** claude-raw-iterative-c1-0
- **Worst:** openai-raw-iterative-6

### Interactive Periodic Table (complex)

- **Avg Score:** -1.0
- **Best:** claude-raw-iterative-c4835-2
- **Worst:** google-sdk-iterative-5

### Real-Time Chat Interface (medium)

- **Avg Score:** 0.0
- **Best:** none
- **Worst:** none

### Stock Ticker Dashboard (simple)

- **Avg Score:** -1.0
- **Best:** claude-sdk-iterative-c1-1
- **Worst:** google-sdk-iterative-5

### Onboarding Wizard (medium)

- **Avg Score:** -1.0
- **Best:** claude-sdk-iterative-c1-1
- **Worst:** google-sdk-iterative-5

## Failures

- **claude-raw-iterative-c1-0** × periodic-table: Claude raw iterative: no compiled code produced
- **claude-raw-iterative-c1-0** × chat-interface: Claude raw iterative: no compiled code produced
- **claude-raw-iterative-c1-0** × stock-ticker: Claude raw iterative: no compiled code produced
- **claude-raw-iterative-c1-0** × onboarding-wizard: Claude raw iterative: no compiled code produced
- **claude-sdk-iterative-c1-1** × periodic-table: Claude SDK iterative: no compiled code produced
- **claude-sdk-iterative-c1-1** × chat-interface: Claude SDK iterative: no compiled code produced
- **claude-raw-iterative-c4835-2** × chat-interface: Timeout after 600000ms
- **claude-sdk-iterative-c4835-3** × periodic-table: Timeout after 600000ms
- **claude-sdk-iterative-c4835-3** × chat-interface: Timeout after 600000ms
- **claude-sdk-iterative-c4835-3** × stock-ticker: Timeout after 600000ms
- **claude-sdk-iterative-c4835-3** × onboarding-wizard: Timeout after 600000ms
- **google-raw-iterative-4** × chat-interface: Google raw iterative: no compiled code produced
- **google-sdk-iterative-5** × chat-interface: Google ADK iterative: no compiled code produced
- **openai-raw-iterative-6** × periodic-table: OpenAI raw iterative: no compiled code produced
- **openai-raw-iterative-6** × chat-interface: OpenAI raw iterative: no compiled code produced
- **openai-raw-iterative-6** × stock-ticker: OpenAI raw iterative: no compiled code produced
- **openai-raw-iterative-6** × onboarding-wizard: OpenAI raw iterative: no compiled code produced
- **openai-sdk-iterative-7** × weather-card: OpenAI SDK iterative: no compiled code produced
- **openai-sdk-iterative-7** × periodic-table: OpenAI SDK iterative: no compiled code produced
- **openai-sdk-iterative-7** × chat-interface: OpenAI SDK iterative: no compiled code produced
- **openai-sdk-iterative-7** × stock-ticker: OpenAI SDK iterative: no compiled code produced
- **openai-sdk-iterative-7** × onboarding-wizard: OpenAI SDK iterative: no compiled code produced
