# Multi-SDK Benchmark Report

**Date:** 2026-03-16T04:34:20.642Z
**Duration:** 52.1s
**Runs:** 15 (13 success, 2 failed)
**Success Rate:** 86.7%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| Anthropic Claude | fast     | n/a       | 35.4s    | $0.0849  | 100%    |
| OpenAI           | premium  | n/a       | 45.1s    | $0.1610  | 100%    |
| OpenAI           | balanced | n/a       | 0ms      | $0.0000  | 0%      |
| Google Gemini    | fast     | n/a       | 40.4s    | $0.0292  | 100%    |

## Variant Rankings (by avg score)

| Rank | Variant                         | Score | Time  | Cost    | Success |
| ---- | ------------------------------- | ----- | ----- | ------- | ------- |
| 1    | openai-sdk-codex-sp             | 0.0   | 0ms   | $0.0000 | 0%      |
| 2    | openai-sdk-codex-iter           | 0.0   | 0ms   | $0.0000 | 0%      |
| 3    | claude-raw-haiku-sp             | n/a   | 30.1s | $0.0774 | 100%    |
| 4    | claude-raw-haiku-iter           | n/a   | 27.0s | $0.0396 | 100%    |
| 5    | claude-sdk-haiku-sp             | n/a   | 45.7s | $0.0914 | 100%    |
| 6    | claude-haiku-mobile-chat        | n/a   | 28.3s | $0.0759 | 100%    |
| 7    | claude-haiku-mobile-fullscreen  | n/a   | 46.8s | $0.1207 | 100%    |
| 8    | claude-haiku-tablet-chat        | n/a   | 29.9s | $0.0756 | 100%    |
| 9    | claude-haiku-tablet-fullscreen  | n/a   | 29.8s | $0.0768 | 100%    |
| 10   | claude-haiku-desktop-chat       | n/a   | 31.3s | $0.0800 | 100%    |
| 11   | claude-haiku-desktop-fullscreen | n/a   | 49.4s | $0.1264 | 100%    |
| 12   | openai-raw-gpt54-sp             | n/a   | 45.1s | $0.1610 | 100%    |
| 13   | google-raw-flash-sp             | n/a   | 51.8s | $0.0670 | 100%    |
| 14   | google-raw-flash-iter           | n/a   | 26.1s | $0.0206 | 100%    |
| 15   | google-sdk-flash-sp             | n/a   | 43.3s | $0.0000 | 100%    |

## Per-Prompt Results

### Weather Card (simple)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-sdk-flash-sp

## Failures

- **openai-sdk-codex-sp** × weather-card: OpenAI SDK single-pass: no compiled code produced
- **openai-sdk-codex-iter** × weather-card: OpenAI SDK iterative: no compiled code produced
