# Multi-SDK Benchmark Report

**Date:** 2026-03-24T06:07:54.428Z
**Duration:** 120.8s
**Runs:** 8 (7 success, 1 failed)
**Success Rate:** 87.5%

## SDK Comparison

| SDK           | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ------------- | -------- | --------- | -------- | -------- | ------- |
| Google Gemini | balanced | 82.2      | 1.1m     | $0.0704  | 88%     |

## Variant Rankings (by avg score)

| Rank | Variant          | Score | Time | Cost    | Success | Tier Eval |
| ---- | ---------------- | ----- | ---- | ------- | ------- | --------- |
| 1    | google-default-0 | 82.2  | 1.1m | $0.0704 | 88%     | 0P/4W/3F  |

## Per-Commit Results

### Weather Card (simple)

- **Avg Score:** 78.0
- **Best:** google-default-0
- **Worst:** google-default-0
- **Tier Eval:** 0 pass, 1 warn, 0 fail

### Multi-Step Survey Form (medium)

- **Avg Score:** 87.6
- **Best:** google-default-0
- **Worst:** google-default-0
- **Tier Eval:** 0 pass, 1 warn, 0 fail

### Kanban Task Board (medium)

- **Avg Score:** -1.0
- **Best:** google-default-0
- **Worst:** google-default-0
- **Tier Eval:** 0 pass, 0 warn, 1 fail

### Interactive Periodic Table (complex)

- **Avg Score:** 0.0
- **Best:** none
- **Worst:** none

### E-Commerce Product Page (complex)

- **Avg Score:** 79.2
- **Best:** google-default-0
- **Worst:** google-default-0
- **Tier Eval:** 0 pass, 1 warn, 0 fail

### Chat Interface (medium)

- **Avg Score:** 83.6
- **Best:** google-default-0
- **Worst:** google-default-0
- **Tier Eval:** 0 pass, 0 warn, 1 fail

### Stock Ticker Dashboard (simple)

- **Avg Score:** 75.0
- **Best:** google-default-0
- **Worst:** google-default-0
- **Tier Eval:** 0 pass, 0 warn, 1 fail

### Onboarding Wizard (medium)

- **Avg Score:** 90.0
- **Best:** google-default-0
- **Worst:** google-default-0
- **Tier Eval:** 0 pass, 1 warn, 0 fail

## Failures

- **google-default-0** × periodic-table: 400 {"error":{"message":"Model generated invalid JSON syntax and the output could not be parsed. Please retry the request. If the issue persists, include this error message in the retry prompt to instruct the model to produce valid JSON.","code":"malformed_tool_call"}}
