# Multi-SDK Benchmark Report

**Date:** 2026-03-17T16:28:26.777Z
**Duration:** 600.0s
**Runs:** 36 (20 success, 16 failed)
**Success Rate:** 55.6%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| Anthropic Claude | balanced | 80.7      | 2.7m     | $0.3585  | 67%     |
| OpenAI           | balanced | 83.6      | 16.7s    | $0.0189  | 17%     |
| Google Gemini    | balanced | 80.4      | 2.6m     | $0.2200  | 83%     |

## Variant Rankings (by avg score)

| Rank | Variant      | Score | Time  | Cost    | Success |
| ---- | ------------ | ----- | ----- | ------- | ------- |
| 1    | openai-raw-2 | 86.6  | 14.2s | $0.0377 | 17%     |
| 2    | google-sdk-5 | 82.5  | 2.4m  | $0.1725 | 83%     |
| 3    | claude-raw-0 | 80.7  | 2.5m  | $0.5345 | 50%     |
| 4    | openai-sdk-3 | 80.6  | 19.2s | $0.0000 | 17%     |
| 5    | google-raw-4 | 78.4  | 2.9m  | $0.2674 | 83%     |
| 6    | claude-sdk-1 | n/a   | 2.9m  | $0.2528 | 83%     |

## Per-Commit Results

### Weather Card (simple)

- **Avg Score:** 83.8
- **Best:** openai-raw-2
- **Worst:** openai-sdk-3

### Kanban Task Board (medium)

- **Avg Score:** 0.0
- **Best:** none
- **Worst:** none

### Interactive Periodic Table (complex)

- **Avg Score:** 86.3
- **Best:** google-sdk-5
- **Worst:** google-raw-4

### Chat Interface (medium)

- **Avg Score:** 77.0
- **Best:** claude-raw-0
- **Worst:** google-sdk-5

### Stock Ticker Dashboard (simple)

- **Avg Score:** 79.5
- **Best:** google-sdk-5
- **Worst:** claude-raw-0

### Onboarding Wizard (medium)

- **Avg Score:** 76.0
- **Best:** google-sdk-5
- **Worst:** google-raw-4

## Failures

- **claude-raw-0** × kanban-board: Claude raw iterative: no compiled code produced
- **claude-raw-0** × periodic-table: Claude raw iterative: no compiled code produced
- **claude-raw-0** × onboarding-wizard: Claude raw iterative: no compiled code produced
- **claude-sdk-1** × kanban-board: Timeout after 600000ms
- **openai-raw-2** × kanban-board: OpenAI raw iterative: no compiled code produced
- **openai-raw-2** × periodic-table: OpenAI raw iterative: no compiled code produced
- **openai-raw-2** × chat-interface: OpenAI raw iterative: no compiled code produced
- **openai-raw-2** × stock-ticker: OpenAI raw iterative: no compiled code produced
- **openai-raw-2** × onboarding-wizard: OpenAI raw iterative: no compiled code produced
- **openai-sdk-3** × kanban-board: OpenAI SDK iterative: no compiled code produced
- **openai-sdk-3** × periodic-table: OpenAI SDK iterative: no compiled code produced
- **openai-sdk-3** × chat-interface: OpenAI SDK iterative: no compiled code produced
- **openai-sdk-3** × stock-ticker: OpenAI SDK iterative: no compiled code produced
- **openai-sdk-3** × onboarding-wizard: OpenAI SDK iterative: no compiled code produced
- **google-raw-4** × kanban-board: Google raw iterative: no compiled code produced
- **google-sdk-5** × kanban-board: Google ADK iterative: no compiled code produced
