<!-- SPDX-License-Identifier: MIT -->
<!-- SPDX-FileCopyrightText: 2025-2026 Marcus Quinn -->

# Chapter 12: Creative Testing and Experimentation Framework

## 12.1 The Scientific Method for Creative

**Step 1: Hypothesis** — "If we [change], then [metric] will [increase/decrease] because [reasoning]."
Example: "If we add customer testimonials to hero images, then CTR will increase 15% because social proof reduces perceived risk."

**Step 2: Test Design** — Control (existing best performer) vs. Variation (single change). Set sample size for statistical significance, duration 3-7 days minimum (accounts for day-of-week effects), and primary success metric.

**Step 3: Execution** — Equal budget, random audience assignment, consistent placement/targeting, clean data collection.

**Step 4: Analysis** — Statistical significance (95% confidence minimum), practical significance (lift magnitude), segment differences, secondary metric impacts.

**Step 5: Implementation** — Scale winners, document insights, plan follow-up tests, share learnings.

## 12.2 Types of Creative Tests

### Element Testing

| Element | Test Dimensions |
|---------|----------------|
| **Headline** | Emotional vs. rational; question vs. statement; length; benefit vs. feature; urgency vs. evergreen |
| **Visual** | Lifestyle vs. product; color palette; talent diversity; background; orientation/cropping |
| **CTA** | Action verb ("Get," "Start," "Try," "Claim"); benefit inclusion; urgency indicators; button color/placement |

### Format Testing

Static vs. video · Single image vs. carousel · Short vs. long-form video · Story vs. feed · Interactive vs. static

### Concept Testing

Problem-solution vs. aspiration · Humor vs. serious · UGC vs. brand-produced · Educational vs. entertainment · Direct response vs. brand storytelling

## 12.3 Test Prioritization Frameworks

**ICE Score = Impact × Confidence × Ease** (each 1-10; prioritize highest)

- **Impact**: Effect on key metrics (10=transformational, 5=meaningful, 1=incremental)
- **Confidence**: Likelihood of success (10=strong data, 5=some evidence, 1=intuition)
- **Ease**: Resource requirements (10=minimal, 5=moderate production, 1=major production)

**RICE Score = (Reach × Impact × Confidence) ÷ Effort** — add Reach for resource-constrained prioritization (10=all audiences, 5=major segments, 1=niche subset).

## 12.4 Sample Size and Statistical Significance

**Required sample size**: Use online calculators with baseline conversion rate, MDE, 80% power, 95% significance.

Rules of thumb: 100 conversions/variation (high-volume) · 50 conversions with larger effect sizes (lower volume) · 10,000+ impressions/variation (brand campaigns).

**Key concepts:**
- 95% confidence = 5% false positive rate (p < 0.05); 99% confidence = 1% (p < 0.01)
- Practical vs. statistical significance: a 2% lift at 99% confidence may not justify production costs; a 50% lift at 90% confidence likely does. Consider both.

## 12.5 Common Testing Pitfalls

| Pitfall | Problem | Solution |
|---------|---------|----------|
| Multiple variables | Can't isolate cause | Test one change at a time, or multivariate with sufficient traffic |
| Ending too early | False conclusions | Pre-determine sample sizes; avoid daily peeking |
| Atypical periods | Holidays/events skew results | Avoid known atypical periods or extend duration |
| Ignoring segments | Overall winner may fail in key segments | Analyze by audience, geography, platform |
| Novelty effects | New creative wins because it's different | Monitor over time — true winners maintain performance |

## 12.6 Building a Testing Culture

**Velocity metrics**: Tests/month · Win rate · Learning rate · Implementation rate · Time to insight

**Testing backlog**: Capture ideas from all team members → score with ICE/RICE → review weekly → archive irrelevant ideas.

**Documentation template:**
```
Test ID / Date / Hypothesis / Variations / Sample size / Results / Winner (confidence) / Learnings / Next steps
```

**Sharing cadence**: Weekly creative reviews · Monthly retrospectives · Quarterly strategy sessions · Internal wiki/knowledge base

## 12.7 Advanced Testing Methodologies

**Sequential Testing**: A vs. Control → winner becomes new Control → repeat. Faster initial insight, less traffic required; slower for comprehensive learning.

**Multi-Armed Bandit**: Algorithmically shifts traffic toward better performers during the test. Reduces opportunity cost; useful for high-traffic/low-risk tests. Requires technical implementation; can mask true performance differences.

**Bayesian Testing**: Uses probability that a variation is best (vs. p-values). Allows continuous monitoring without p-hacking concerns; more intuitive for business decisions. Available in VWO and Optimizely.

## 12.8 Testing Program Maturity

| Level | Name | Characteristics |
|-------|------|-----------------|
| 1 | Ad Hoc | Intuition-driven, no process, limited docs, results often ignored |
| 2 | Structured | Regular cadence, basic docs, hypothesis-driven, results inform some decisions |
| 3 | Systematic | Comprehensive roadmap, statistical rigor, cross-functional, insights drive strategy |
| 4 | Predictive | AI/ML optimization, automated test generation, predictive modeling, continuous autonomous optimization |
