# Multi-SDK Benchmark Report

**Date:** 2026-03-18T06:35:20.824Z
**Duration:** 900.1s
**Runs:** 72 (66 success, 6 failed)
**Success Rate:** 91.7%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| OpenAI           | balanced | 83.0      | 22.8s    | $0.0543  | 88%     |
| Google Gemini    | balanced | 76.3      | 2.3m     | $0.3689  | 96%     |
| Anthropic Claude | balanced | 75.3      | 5.0m     | $0.7533  | 92%     |

## Variant Rankings (by avg score)

| Rank | Variant             | Score | Time  | Cost    | Success |
| ---- | ------------------- | ----- | ----- | ------- | ------- |
| 1    | openai-raw-c0437-2  | 84.2  | 22.5s | $0.0986 | 83%     |
| 2    | claude-sdk-c1-9     | 84.0  | 3.0m  | $0.2746 | 83%     |
| 3    | openai-raw-c1-0     | 83.9  | 23.4s | $0.1294 | 83%     |
| 4    | openai-sdk-c0437-3  | 82.4  | 20.8s | $0.0000 | 100%    |
| 5    | openai-sdk-c1-1     | 81.6  | 24.7s | $0.0000 | 83%     |
| 6    | google-raw-c1-4     | 78.9  | 2.2m  | $0.3314 | 100%    |
| 7    | google-sdk-c3975-7  | 77.8  | 2.3m  | $0.3699 | 100%    |
| 8    | google-sdk-c1-5     | 75.4  | 2.1m  | $0.2999 | 83%     |
| 9    | claude-raw-c1-8     | 74.6  | 2.3m  | $0.6300 | 100%    |
| 10   | claude-raw-c4835-10 | 74.5  | 8.9m  | $1.2049 | 83%     |
| 11   | google-raw-c3975-6  | 73.1  | 2.7m  | $0.4628 | 100%    |
| 12   | claude-sdk-c4835-11 | n/a   | 6.3m  | $0.8991 | 100%    |

## Per-Commit Results

### Weather Card (simple)

- **Avg Score:** 80.0
- **Best:** openai-raw-c0437-2
- **Worst:** google-sdk-c1-5

### Multi-Step Survey Form (medium)

- **Avg Score:** 76.3
- **Best:** openai-raw-c1-0
- **Worst:** google-raw-c1-4

### Interactive Periodic Table (complex)

- **Avg Score:** 73.9
- **Best:** google-sdk-c3975-7
- **Worst:** claude-raw-c4835-10

### Chat Interface (medium)

- **Avg Score:** 78.4
- **Best:** openai-raw-c1-0
- **Worst:** google-sdk-c3975-7

### Stock Ticker Dashboard (simple)

- **Avg Score:** 83.2
- **Best:** openai-raw-c1-0
- **Worst:** google-sdk-c3975-7

### Onboarding Wizard (medium)

- **Avg Score:** 80.5
- **Best:** openai-sdk-c0437-3
- **Worst:** claude-raw-c1-8

## Failures

- **openai-raw-c1-0** × onboarding-wizard: OpenAI raw iterative: no compiled code produced (1 turns, 42094 tokens)
  No tool calls made — model responded with text only
- **openai-sdk-c1-1** × onboarding-wizard: OpenAI SDK iterative: no compiled code produced
  No tool calls made — model responded with text only
- **openai-raw-c0437-2** × chat-interface: OpenAI raw iterative: no compiled code produced (1 turns, 45309 tokens)
  No tool calls made — model responded with text only
- **google-sdk-c1-5** × onboarding-wizard: Google ADK iterative: no compiled code produced
- **claude-sdk-c1-9** × onboarding-wizard: Claude SDK iterative: no compiled code produced (190.0s elapsed, 18 turns)
  sourceCode captured: false, meta captured: false
- **claude-raw-c4835-10** × onboarding-wizard: Timeout after 900000ms
