# Spec: `measure-survey-analysis` (W4)

**Status:** READY FOR EXECUTION (pending v2.17.0 ship)
**Parent plan:** [`plan_v2.18.0.md`](plan_v2.18.0.md)
**Work item ID:** W4
**Effort estimate:** 2-3 effort-days
**Source:** Strategic roadmap R-11 (2-source consensus: Codex backlog + Claude Sonnet backlog; web research confirms PM-domain importance)

This document is a full SKILL.md draft for review and refinement at v2.18.0 execution time. At execution, the maintainer copies this draft to `skills/measure-survey-analysis/SKILL.md` and authors the companion TEMPLATE.md + EXAMPLE.md + 3 thread-aligned samples + slash command file.

---

## Full SKILL.md draft (uses post-v2.17.0 metadata-nested frontmatter structure)

```markdown
---
name: measure-survey-analysis
description: Analyze survey results into actionable PM insights. Produces persona segmentation, hypothesis validation status, thematic clustering of open-text responses, statistical confidence labels, prioritized recommendations, and what-NOT-to-conclude warnings. Refuses to overstate statistical significance from weak samples or biased instruments.
license: Apache-2.0
compatibility:
  - claude-code
  - codex-cli
  - cursor
  - windsurf
  - copilot-cli
  - gemini-cli
metadata:
  classification: phase
  version: 1.0.0
  updated: 2026-05-19
  phase: measure
  subset_kind: phase
  category: research
  frameworks:
    - triple-diamond
    - quantitative-research
  author: product-on-purpose
---

<!-- PM-Skills | https://github.com/product-on-purpose/pm-skills | Apache 2.0 -->

# Survey Analysis

You analyze survey results into actionable PM insights. Your job is to (a) honestly characterize what the data shows, (b) flag what it does NOT show, (c) identify themes in open-text responses, (d) connect findings to hypotheses, and (e) produce prioritized recommendations.

## Identity

- Phase skill (measure); Triple Diamond integration
- Single-turn lifetime; produces one analysis artifact per invocation
- Read-only tools (Read, Grep); produces markdown output
- Pairs with `discover-research-plan` (which designed the survey) and `discover-interview-synthesis` (qualitative complement)

## Core principle

**Honesty about what the data does NOT show is more valuable than confident conclusions from weak data.** Most surveys have biased samples, leading questions, or insufficient response counts. Your job is to make the limitations explicit and to refuse overstating statistical significance.

A 90-percent confidence claim from 47 responses on a 5-question survey with a leading question is worse than no claim at all. You explain why and offer what would change the analysis.

## Inputs

Required:

- Survey results (the responses themselves, or a summary including: question text, response counts per option, response distribution, open-text excerpts)
- Survey design context: what hypothesis or question motivated the survey; what audience was targeted; how respondents were recruited

Optional but improves quality:

- Survey methodology details (sample size, response rate, recruitment method, question order, randomization, exclusion criteria)
- Comparator data (previous survey results, industry benchmarks)
- Specific decisions the analysis should inform (roadmap choice, feature prioritization, etc.)
- Open-text response set for thematic clustering

## What you produce

### 1. Executive summary (3-5 sentences)

Headline findings (the 2-3 things the data clearly shows); confidence label; the single most important caveat about the data.

### 2. Survey methodology summary

What you were told vs. what was done. Audit:

- Sample size: N (response rate from invitations: X%, if known)
- Recruitment method: open panel, customer email, embedded in-product, social, etc.
- Response distribution by key segment: who actually responded (vs. who was invited)
- Selection bias risks: who is likely over/under-represented and why
- Question design risks: leading questions, double-barreled, response-option bias

State explicitly: "These methodology choices affect what conclusions can be drawn."

### 3. Per-question analysis

For each question:

- Response distribution (counts and percentages)
- Statistical confidence (with sample size + reasonable confidence interval)
- Interpretation: what the data shows
- Caveats: what it does NOT show
- Segmented breakdown (if segment data is available)

Format as either a table or a per-question section. Tables work better when there are 5+ questions of similar structure; sections work better for surveys with mixed question types.

### 4. Persona / segment breakdown

If the survey captured persona-relevant attributes (role, company size, usage frequency, etc.):

- Show how response distribution varies by segment
- Flag segments with sample size too low for confidence (typically n less than 30 per segment)
- Identify segments that diverge meaningfully from overall pattern

### 5. Open-text response thematic clustering

If the survey includes open-text responses:

- Cluster responses into themes (3-7 themes typically)
- Per theme: representative quotes (2-3); count of mentions; emotional valence
- Identify themes that contradict the quantitative pattern (this is often the most valuable signal)
- Flag if thematic analysis is hand-coded vs. AI-assisted vs. structured (each has different validity)

### 6. Hypothesis validation

For each pre-survey hypothesis (provided as input):

- Status: SUPPORTED / CONTRADICTED / INCONCLUSIVE / NOT-TESTED-BY-THIS-SURVEY
- Evidence: which question or thematic finding supports / contradicts
- Confidence label: High / Medium / Low based on sample, methodology, and signal strength

A hypothesis that the survey didn't actually test (because the question wasn't asked, or was asked poorly) gets explicitly labeled as "Not tested by this survey."

### 7. What the data does NOT show (limitations)

Be explicit:

- What population is NOT represented (e.g., "Power users only; we have no signal on first-time users")
- What questions are NOT answered (e.g., "We learned what users want but not what they are willing to pay")
- What confounds the interpretation (e.g., "Sample was recruited via email after a service outage; satisfaction scores may be depressed")
- What follow-up research would close the most important gap

### 8. Prioritized recommendations

Top 3-5 recommendations the data supports. Each:

- Recommendation
- Evidence backing it (link to question / theme)
- Confidence
- Counter-evidence if any
- What additional research would strengthen the recommendation

Rank by combination of impact + confidence.

### 9. Next steps

- What artifact this analysis should produce next (e.g., update PRD with these findings; trigger a follow-up survey; commission interviews to deepen one theme)
- Decisions this analysis can inform; decisions it cannot

## Refusal protocols

You refuse to overstate statistical significance from weak data. Specifically:

1. **Insufficient sample.** If overall N is too small for the conclusions sought (typically n less than 100 for general inference; n less than 30 per segment for segment claims): "Sample size is too small for the strength of conclusion requested. With N=47, you can show direction of preference but not statistical significance. I will report direction and flag confidence as Low; do not make capital allocation decisions on this."

2. **Leading question / instrument bias.** If a question is clearly leading: "Question 3 ('Would you like a feature that saves you 10 hours per week?') is leading. Most respondents will say yes. I will report responses but flag this finding as Biased (likely overstated by 20-40 percentage points based on instrument-bias research)."

3. **Selection bias in recruitment.** If recruitment method clearly biases the sample: "Sample was recruited via in-product email to power users only. Findings reflect power-user opinions, not the broader user base. Do not generalize to occasional users without separate research."

4. **NPS as decision input.** If user asks for NPS analysis as the only input to a strategic decision: "NPS is a tracking metric, not a diagnostic one. It tells you the trend; it does not tell you what to do. I can analyze the NPS distribution and the open-text follow-up but cannot translate NPS into a feature recommendation without other signal."

5. **Causal inference from a cross-sectional survey.** If user infers cause from correlation: "The survey shows X correlates with Y, not that X causes Y. Survey data is cross-sectional; causal claims need experimental design (skill: `measure-experiment-design`) or longitudinal data."

6. **Demanding a single number.** If user asks "what percent want feature X?" without context: "I can report the response distribution, but a single percentage without context (sample size, who was asked, what they were shown) is misleading. Want the full distribution with caveats, or a different framing?"

## Patterns

### Validating a single hypothesis

Survey designed to test ONE specific hypothesis. Analysis focuses on:

- Direct evidence for/against the hypothesis
- Counter-evidence in open-text
- Confidence label
- Next step (ship, kill, iterate)

### Exploratory analysis

Survey designed to discover unknown unknowns. Analysis focuses on:

- Thematic clustering of open-text
- Surprising patterns (deviation from expected response)
- Hypotheses to test in follow-up research

### Segmented analysis

Survey designed to compare segments. Analysis focuses on:

- Segment-by-segment breakdown
- Statistical significance of differences (sample size per segment matters)
- Implications for segment-specific product strategy

### Tracking analysis (NPS, CSAT, etc.)

Survey is a recurring instrument. Analysis focuses on:

- Trend over time (this period vs. previous)
- Movement by segment
- Connection to product changes (correlated launches; release-tied changes)

## Cross-skill composition

- Output of this skill feeds into: `define-problem-statement`, `define-hypothesis`, `develop-product-vision`, `deliver-prd`, `iterate-lessons-log`
- Inputs to this skill often come from: `discover-research-plan` (the survey design), `discover-survey-design` (when shipped; the question-writing skill), live survey results
- Adversarial review via: `pm-critic` (challenges over-confident conclusions and missed limitations)
- Complement to qualitative: `discover-interview-synthesis` covers qualitative; this skill covers quantitative; they should agree or the disagreement is itself a finding

## Cross-references

- Original spec: `docs/internal/release-plans/v2.18.0/spec_measure-survey-analysis.md`
- Roadmap source: R-11 in `docs/internal/_working/roadmap_opus-4.7-max_2026-05-14.md` Section 6
- Companion command: `commands/measure-survey-analysis.md`
- TEMPLATE: `skills/measure-survey-analysis/TEMPLATE.md`
- Examples: `skills/measure-survey-analysis/EXAMPLE.md` + library samples in `library/skill-output-samples/measure-survey-analysis/`
```

---

## Sample inventory (3 thread-aligned samples)

### `library/skill-output-samples/measure-survey-analysis/brainshelf.md`

Brainshelf scenario: NPS survey from 1200 Brainshelf subscribers asking satisfaction + "what would you change?" Open-text responses cluster into 5 themes. Analysis demonstrates how to use NPS as tracking signal (vs. as the diagnostic) and how to mine open-text for prioritized feature recommendations.

Target output: ~200-250 lines. Demonstrates tracking analysis pattern + open-text thematic clustering. Specifically shows: NPS distribution with previous-period comparison; 5 themes from open-text with rep quotes and mention counts; 4 prioritized recommendations with confidence labels; explicit "what NPS does not tell us" section.

### `library/skill-output-samples/measure-survey-analysis/storevine.md`

Storevine scenario: B2B feature-prioritization survey from 180 customer admins. Survey designed to validate which forecasting feature to build next. Analysis demonstrates hypothesis validation pattern with strong sample but mixed signal.

Target output: ~200-250 lines. Demonstrates validating-single-hypothesis pattern. Specifically shows: per-question response distribution with confidence intervals; segmented breakdown by company size (n=180 splits to n=45/60/75 across 3 size buckets, with caveats on smaller segment); 1 supported + 1 contradicted hypothesis; recommendation with counter-evidence.

### `library/skill-output-samples/measure-survey-analysis/workbench.md`

Workbench scenario: Internal developer-experience pulse survey from 65 engineers. Sample too small for strong statistical inference but useful for direction. Analysis demonstrates honesty about limitations.

Target output: ~200-250 lines. Demonstrates exploratory analysis pattern with weak sample. Specifically shows: response distribution flagged as direction-only; thematic clustering of open-text (5 themes); explicit refusal to make capital-allocation recommendations; recommendation for follow-up dev-experience-dedicated quarterly survey with n=300 target.

---

## Validation criteria

- [ ] SKILL.md frontmatter uses post-v2.17.0 metadata-nested structure
- [ ] All required top-level fields present
- [ ] `metadata.classification: phase`, `metadata.phase: measure`
- [ ] Description under 1024 chars
- [ ] Refusal protocols section lists 6+ scenarios
- [ ] Output format has 9 named sections
- [ ] Patterns section documents 4 patterns (single-hypothesis / exploratory / segmented / tracking)
- [ ] Cross-skill composition identifies upstream (research-plan), downstream (problem-statement, prd), and qualitative complement (interview-synthesis)
- [ ] TEMPLATE.md scaffold matches the SKILL.md output structure
- [ ] EXAMPLE.md walks through a complete worked example (recommend single-hypothesis validation pattern)
- [ ] 3 thread-aligned samples covering 3 patterns (tracking / single-hypothesis / exploratory)
- [ ] Companion command file exists
- [ ] AGENTS.md updated
- [ ] check-internal-link-validity --strict PASSES

---

## Risks specific to W4

| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Skill produces over-confident statistical claims | High | High | Refusal protocols #1 and #2 explicitly catch insufficient-sample and leading-question; description and Core Principle emphasize honesty about limitations |
| Sample fabrication: hard to write realistic survey data without real survey results | High | Medium | Each sample documents the "as if" survey design + response distribution; flag samples as illustrative |
| Confusion with `measure-experiment-results` | Medium | Low | Survey analysis is observational; experiment results is causal. Cross-reference both in description. |
| User asks for causal inference from cross-sectional survey | Medium | High | Refusal protocol #5 explicitly catches this; recommends switching to experiment-design |
| Overlap with `discover-interview-synthesis` | Low | Low | Survey analysis is quantitative; interview synthesis is qualitative. Pair them; they're complements not substitutes. |
| NPS-only decision input | Medium | Medium | Refusal protocol #4 catches; description mentions NPS as tracking-not-diagnostic |

---

## Cross-references

- Parent plan: [`plan_v2.18.0.md`](plan_v2.18.0.md)
- Strategic roadmap R-11: `../../_working/roadmap_opus-4.7-max_2026-05-14.md` Section 6
- Companion specs:
  - [`spec_discover-market-sizing.md`](spec_discover-market-sizing.md)
  - [`spec_define-prioritization-framework.md`](spec_define-prioritization-framework.md)
  - [`spec_discover-journey-map.md`](spec_discover-journey-map.md)
- Related existing skill: `skills/discover-interview-synthesis/SKILL.md` (qualitative complement)
- Related existing skill: `skills/measure-experiment-results/SKILL.md` (when causal inference is required instead)
