# Frame Detection (Connected Components)

Phase D of the spritesheet pipeline. Connected-components clustering replaces naive grid-math cropping. The article's key technical insight: generated frames drift within their cells, so cell-bbox crops capture neighbor pixels and truncate the current sprite.

## The algorithm

```
1. Load the raw spritesheet PNG (RGBA, magenta bg).
2. Build a binary mask: True for non-magenta pixels (chroma threshold T), False otherwise.
3. Run 2-connectivity (4-neighbor) flood fill from each unvisited True pixel.
4. Each flood-fill pass forms one cluster (component).
5. For each component:
   a. Compute its pixel-extent bounding box.
   b. Reject components below MIN_COMPONENT_PIXELS (filter fragments).
6. Sort components by (top, left) — top-to-bottom, left-to-right reading order.
7. Map components to grid cells by their centroid → cell-index lookup.
```

The output is one cropped PNG per component, named by its target grid index (`<name>_frame_00.png`, `_frame_01.png`, ...).

## Why connected components?

Grid math:
```
frame = sheet.crop((c*CELL, r*CELL, (c+1)*CELL, (r+1)*CELL))
```

This breaks because:

1. The model places frames slightly off-cell — drift is typically ±5-15% of cell size.
2. A drifted character's edge falls inside the next cell's bbox, contaminating the neighbor.
3. The current cell's edge crops the character's silhouette mid-pixel.

Connected components find the actual character in pixel space, regardless of its cell-relative position.

## Pseudocode

```python
import numpy as np
from PIL import Image

def extract_frames(
    sheet_path: Path,
    grid_cols: int,
    grid_rows: int,
    chroma_color: tuple[int, int, int] = (255, 0, 255),
    chroma_threshold: int = 30,
    min_component_pixels: int = 200,
) -> list[Image.Image]:
    """Extract frames via connected-components clustering."""
    img = Image.open(sheet_path).convert("RGBA")
    arr = np.array(img)

    # Binary mask: True where pixel is NOT chroma color (within threshold)
    chroma = np.array(chroma_color)
    diff = np.abs(arr[..., :3].astype(int) - chroma).sum(axis=-1)
    mask = diff > chroma_threshold

    # 4-connected component labeling
    labels = label_connected_components(mask)  # custom or scipy.ndimage.label

    components: list[Image.Image] = []
    for label_id in range(1, labels.max() + 1):
        ys, xs = np.where(labels == label_id)
        if len(ys) < min_component_pixels:
            continue  # filter fragments

        top, bot = ys.min(), ys.max() + 1
        left, right = xs.min(), xs.max() + 1
        crop = img.crop((left, top, right, bot))
        components.append(crop)

    # Sort top-to-bottom, left-to-right by bounding-box top-left
    components.sort(key=lambda c: (
        c.getbbox()[1] if c.getbbox() else 0,
        c.getbbox()[0] if c.getbbox() else 0,
    ))

    expected = grid_cols * grid_rows
    if len(components) != expected:
        raise FrameCountMismatchError(
            f"detected {len(components)} components, grid expected {expected}"
        )

    return components
```

`scipy.ndimage.label` is the preferred connected-components implementation. If scipy is not installed, the pure-Python fallback (BFS queue per seed pixel) is ~10x slower but correct.

## Grid density limits

Connected-components extraction is only as good as the source image. Image-gen models (Codex, Nano Banana, etc.) do not respect grid boundaries when the per-cell pixel budget shrinks — characters drift across cell edges, frames get clipped, and the model often draws the same character at different scales between cells. Beyond a certain density, no post-processor can recover usable frames.

| Grid | Total frames | Cell @ 1024px canvas | Reliability | Notes |
|---|---|---|---|---|
| 4×4 | 16 | 256×256 | Reliable | Plenty of cell budget; characters stay within bounds. |
| 8×8 | 64 | 128×128 | Borderline | Use `--cell-aware` connected-components and `--reject-edge-touch`. Expect occasional drift. |
| 16×16 | 256 | 64×64 | NOT recommended | Image-gen models cannot keep characters within 64-pixel cells. Live-demo evidence below. |

The pipeline `sprite_pipeline.py` emits a stderr warning when `rows * cols > 64` and continues — the user can suppress with `--confirm-dense-grid`, but the warning exists for a reason.

### Live-demo evidence (April 2026)

In the demo at `<your-output-dir>/` (author's local validation harness, replace before use), asset 04 (`gameboy-tiny-walk`, 16×16 = 256 frames at 64px) was generated by Codex CLI image_gen against an explicit 16×16 grid prompt. Observed failures in the raw output:

- The model produced a single small character in the center of the canvas instead of 256 individual frames.
- Naive grid-extract carved 256 boxes out of the canvas; only the centermost ~6-9 boxes contained any character pixels at all.
- The character that did appear had a magenta box drawn around it (the model interpreted "16x16 grid of cells" as "draw a 16x16 outlined box around your single character").
- Bg removal then converted the magenta box-outline to transparency, but the character was so small inside its surviving cell that the asset was effectively unusable.

That asset was downgraded to **8×8 = 64 frames at 128px** for the final demo, where the model produced reliable per-cell frames. The 16×16 path remains accessible via `--confirm-dense-grid` for experimentation, but it should be considered a research-only path.

### Practical guidance

- **Default to 4×4** for most action loops (walk, attack, idle). 16 frames is plenty for a smooth animation at 10fps.
- **8×8 max** when you genuinely need 64 frames (multi-direction × multi-state). Use 128px cells minimum.
- **Use 16×16+** only when you have a hard requirement and can manually validate the output.
- **Cell size is a quality knob:** doubling cell size from 64 to 128 quadruples the per-cell pixel budget for the model. Prefer fewer larger cells over more smaller cells.

## Tuning parameters

| Parameter | Default | Adjust if |
|-----------|---------|-----------|
| `chroma_threshold` | 30 (RGB sum-of-abs-diff) | Magenta fringing → increase; legitimate magenta-adjacent pixels lost → decrease |
| `min_component_pixels` | 200 | Detection picks up fragments (eyes, glints) → increase to ~500; small details lost → decrease |
| `connectivity` | 4-neighbor | Components fragment on diagonal-only links → use 8-neighbor (`scipy.ndimage.label(structure=np.ones((3,3)))`) |

The skill exposes these as flags:

```bash
python3 sprite_process.py extract-frames \
    --input sheet.png \
    --grid 4x4 \
    --chroma-threshold 30 \
    --min-pixels 200 \
    --output-dir frames/
```

## Failure modes

### Components merge across cells

**Symptom:** detected count < grid count. Two adjacent characters got connected by a thin chroma-failed pixel bridge.

**Cause:** Magenta fringing too aggressive (low threshold lets near-magenta pixels survive), OR cell separation gap is too small in the canvas template.

**Fix:** Increase `--chroma-threshold` (relaxes the bg mask, drops more fringe). If that fails, regenerate the canvas with larger gutters between cells (edit `sprite_canvas.py`'s `border-width` constant).

### Components fragment within cells

**Symptom:** detected count > grid count. One character split into multiple components (head + body separated, glow aura detected as separate component).

**Cause:** Aggressive chroma-threshold has bitten into the character's color, OR the character has a low-saturation segment that matches magenta.

**Fix:** Decrease `--chroma-threshold` (tighter bg mask), OR increase `--min-pixels` to filter fragments, OR use 8-neighbor connectivity.

### Detected count = expected, but frames are wrong

**Symptom:** count matches, but `frame_03` shows the character that should be in cell 5.

**Cause:** Component sort order does not match grid reading order, OR multiple components per cell got confused.

**Fix:** Inspect the bounding-box centroids of components. The sort key is `(top, left)` — if frames have nearly-equal `top` values, the sort is unstable. Use cell-centroid mapping instead: each component's centroid is mapped to the nearest cell (col, row), and the component is assigned that cell's frame index.

## Centroid-to-cell mapping (alternative sort)

When frame-order matters and frames drift significantly:

```python
def assign_to_cells(components, grid_cols, grid_rows, sheet_w, sheet_h):
    cell_w = sheet_w / grid_cols
    cell_h = sheet_h / grid_rows
    assignments = {}
    for comp in components:
        cy = (comp.bbox.top + comp.bbox.bottom) / 2
        cx = (comp.bbox.left + comp.bbox.right) / 2
        col = min(int(cx / cell_w), grid_cols - 1)
        row = min(int(cy / cell_h), grid_rows - 1)
        idx = row * grid_cols + col
        if idx in assignments:
            # collision: prefer larger component
            if comp.area > assignments[idx].area:
                assignments[idx] = comp
        else:
            assignments[idx] = comp
    return [assignments.get(i) for i in range(grid_cols * grid_rows)]
```

This is the implementation in `sprite_process.py extract-frames` when `--cell-aware` is passed (default for grids ≥ 2x2).

## Output

Each component is saved as `<output-dir>/<name>_frame_NN.png`. The component's PNG dimensions equal its bounding-box size — frames are not yet rescaled to a uniform size; that happens in Phase F (anchor-alignment.md).

A metadata sidecar is written: `<output-dir>/frame_metadata.json`:

```json
{
  "sheet": "sheet.png",
  "grid": [4, 4],
  "components": [
    {"index": 0, "bbox": [12, 8, 240, 252], "area": 23440, "centroid": [126, 130]},
    {"index": 1, "bbox": [268, 12, 510, 256], "area": 22120, "centroid": [389, 134]},
    ...
  ],
  "rejected": 2,
  "warnings": []
}
```

`rejected` counts components below `--min-pixels`. Useful for tuning thresholds.

<!-- no-pair-required: section header; pair lives in subsection -->
## Signals and Fixes

### Signal: Naive grid math for frame extraction

**What it looks like:**
```python
for r in range(rows):
    for c in range(cols):
        frame = sheet.crop((c * cell, r * cell, (c+1) * cell, (r+1) * cell))
```

**Why wrong:** Generated frames drift ±5-15% of cell size. Naive crops capture neighbor pixels and truncate the current sprite. Output frames look ragged at the edges and contain ghosted neighbor silhouettes.

**Do instead**: Connected-components clustering with chroma-key masking. Each component is bounded by the actual character pixel extent, not the cell math. The cell index is recovered post-hoc from component centroid.

## Cell pitch is derived from raw_size, never assumed from final canvas size

**Rule: Cell size is derived from raw_size and grid, never assumed from final canvas size.**

This is the load-bearing invariant for the naive-grid extractor. The bug it
prevents: when an image-gen backend (Codex, Nano Banana, etc.) returns a raw
sheet whose dimensions are NOT `cols * cell_size`, the natural sprite pitch
in the source is `raw_size / cols` (which is usually fractional), not
`cell_size`. A whole-image LANCZOS resize from raw to canonical canvas does
NOT relocate sprite centers onto the cell_size pitch — it only resamples
their pixels. The sprites stay at their natural-pitch positions, and a
cell_size grid then cuts every character mid-body.

Live evidence (April 2026): asset 05 (`05-nes-powerhouse-attack`) in the
public demo. Codex returned a 1254×1254 raw for an 8×8 grid prompt at
target cell_size=128. The pre-fix code resized 1254→1024 with LANCZOS and
sliced at 128. Result: every character cropped at the arms or feet,
animation stuttered, frame strip showed off-grid characters. The fix
derives the pitch from raw_size: 1254 / 8 = 156.75 px per cell. Each cell
is cropped at the natural pitch and resampled to 128. The new final-sheet
shows characters centered inside their cells.

### Hybrid algorithm (`sprite_process.slice_grid_cells`)

```python
def slice_grid_cells(sheet, cols, rows, cell_size):
    raw_w, raw_h = sheet.size
    canonical_w = cols * cell_size
    canonical_h = rows * cell_size

    # Case 1 — exact match: direct slice, no resample (zero-waste happy path).
    if raw_w == canonical_w and raw_h == canonical_h:
        return [sheet.crop((c * cell_size, r * cell_size,
                            (c + 1) * cell_size, (r + 1) * cell_size))
                for r in range(rows) for c in range(cols)]

    # Case 2 & 3 — derive pitch from raw size, slice at natural pitch,
    # resample each cell to cell_size. Float pitch + rounded boundaries
    # absorbs sub-pixel rounding into the per-cell crop, not the pitch.
    pitch_x = raw_w / cols
    pitch_y = raw_h / rows
    cells = []
    for r in range(rows):
        for c in range(cols):
            x0, y0 = round(c * pitch_x), round(r * pitch_y)
            x1, y1 = round((c + 1) * pitch_x), round((r + 1) * pitch_y)
            crop = sheet.crop((x0, y0, min(x1, raw_w), min(y1, raw_h)))
            if crop.size != (cell_size, cell_size):
                crop = crop.resize((cell_size, cell_size), Image.Resampling.LANCZOS)
            cells.append(crop)
    return cells
```

Why this is hybrid: the algorithm picks the cheapest correct path for the
input. Exact-match canvases skip the resample entirely (case 1). Fractional
canvases pay one LANCZOS resample per cell (cases 2–3) but never resample
the WHOLE sheet — which is the operation that loses sprite-pitch alignment.

### Synthesized-canvas test pattern

`skills/game/game-sprite-pipeline/scripts/test_slice_grid_cells.py` synthesizes
magenta canvases with green-marker disks placed at known cell centers, runs
each through `slice_grid_cells`, and asserts each extracted cell's marker
centroid lies within `cell_size / 8` of the cell center. Cases:

| Canvas | Grid | Cell | Pitch | Path |
|---|---|---|---|---|
| 1254×1254 | 8×8 | 128 | 156.75 | fractional (case 3) — the asset 05 reproduction |
| 1024×1024 | 8×8 | 128 | 128.00 | exact (case 1) |
| 1280×1280 | 8×8 | 128 | 160.00 | integer (case 2) |
| 1024×1024 | 2×2 | 512 | 512.00 | exact (case 1) — portrait-loop |
| 1100×1100 | 2×2 | 512 | 550.00 | fractional (case 3) |
| 1024×1024 | 4×4 | 256 | 256.00 | exact (case 1) — action loop |
| 1535×1535 | 4×4 | 256 | 383.75 | fractional (case 3) |

All seven cases pass. The test is pure deterministic Python (PIL + numpy),
no LLM, no external API. Run via:

```bash
python3 skills/game/game-sprite-pipeline/scripts/test_slice_grid_cells.py
```

This is the deterministic gate the toolkit philosophy demands — algorithms
verified by code, not eyeballs.

### v8 mega-component fallback (asset 19 painted-veteran)

When the chroma threshold is loose enough that the painted-style rim
light bridges adjacent cells, `extract_components` returns one giant
component spanning most of the sheet. Concrete reproduction (April 2026):
asset 19 (`19-veteran-running-to-ropes`, slay-the-spire-painted, 4x4 grid
at 256px) generated component 0 with bbox=(0,0,1024,1024) and area
550,888 pixels. The centroid mapping then assigned that mega-blob to
ONE cell (cell 9, centroid at (499,527)), leaving the other 15 cells
empty. Final-sheet had 1 cell of content, 15 blanks.

Fix: post-extraction, check whether any single component has area
> cell_area * 1.5 (a single component covering more than 1.5x a cell is
the signature of cross-cell bridging). If so, abandon the components
path and use per-cell despill via `slice_grid_cells`. Implementation
in `<your-output-dir>/generate.py post_process_spritesheet` (search for
"mega_component_detected").

The per-cell path uses a tighter pass2 threshold (60 instead of the
default 90) for painted-style assets so the despill chain doesn't bridge
within cells either. Pixel-art styles (NES, SNES, GB, Genesis) keep the
default 90 because their backgrounds are flat solid magenta.

### Content-aware slicing (`slice_with_content_awareness`)

When Codex paints content that legitimately extends past the conceptual cell
boundary — dragon flame breath that trails 30-50 px past the cell edge (asset
27), plasma trails that cross every internal vertical boundary (asset 30),
extended limbs that reach into the neighbor (boss-character punches) — the
strict-pitch slicer cuts that content at the boundary, dropping the trailing
portion AND pasting it onto the neighbor cell where it has no anchor.

`slice_with_content_awareness` is the alternative slicer for assets where
this happens. The `has_effects: True` (or explicit `content_aware_extraction:
True`) flag in the asset spec switches the pipeline from `slice_grid_cells`
to this function.

**Algorithm (per cell at fractional pitch):**

1. Identify the cell's natural rectangle `[x0, y0, x1, y1]` at `raw_w/cols` ×
   `raw_h/rows` pitch.
2. For each cell-edge: count non-magenta pixels in an `edge_band_px`-wide
   strip just INSIDE the cell boundary AND a same-width strip just OUTSIDE
   (in neighbor territory).
3. If both strips have ≥ `min_edge_run` (default 8) continuous non-magenta
   pixels, content extends across the boundary. Expand the cell rectangle
   outward up to `max_expansion_pct` of the cell pitch (default 0.30 = 30%).
4. The expansion stops at the FIRST column/row in the neighbor that drops
   back to magenta (the natural tail of the effect), or at `max_expansion_pct`
   — whichever comes first.
5. Resample the expanded crop to `cell_size × cell_size` (same shape contract
   as `slice_grid_cells`).

**Centroid-ownership invariant.** When cell A's right boundary expands into
B's left territory, the slicer does NOT double-count: components are assigned
to the cell that contains their mass-centroid. B sees only the body of B
(centered on its centroid) without the trail that belongs to A. Beyond
`max_expansion_pct=0.30` the content has crossed the centroid and now belongs
to the next cell anyway.

**No-op for clean assets.** If neither inside nor outside strip has the
`min_edge_run` of non-magenta pixels at any cell edge, the slicer reduces to
the strict pitch behavior. Pixel-art assets (NES, SNES, GB) where the model
respects cell boundaries get the same output as `slice_grid_cells`.

**Where wired in.** `<your-output-dir>/generate.py post_process_spritesheet`
checks `spec.get("content_aware_extraction") or spec.get("has_effects")` and
dispatches accordingly. Implementation in `sprite_slicing.slice_with_content_awareness`
(`scripts/sprite_slicing.py:344`). Module location updated per ADR-205
(sprite_process split into 5 cohesive modules); also re-exported from
`sprite_process` for backward compat with `from sprite_process import
slice_with_content_awareness` callers.

**Caveat (ADR-207 RC-1, dense-grid downgrade).** On dense grids
(`cols * rows >= 16` AND both `cols >= 4` and `rows >= 4` — i.e. 4x4 and
denser), the slicer dispatch in `sprite_pipeline.py` and `cmd_extract_frames`
silently downgrades `--content-aware-extraction` (or
`content_aware_extraction: True` in the spec) to strict-pitch slicing with a
WARNING log. The downgrade exists because content-aware routing on dense
Codex fractional-pitch raws drops cells via centroid drift (RC-1). To
explicitly opt INTO content-aware on a dense grid (legitimate sparse
effects assets — fire breath, plasma trails, projectile auras) pass the
new `--effects-asset` flag. Sparse grids (3x3 and below, or with a
degenerate dim like 1xN) keep content-aware as a free choice.

### Where the cell-pitch rule applies

Every consumer that takes a raw multi-cell sheet and slices it into
fixed-size cells:

- `skills/game/game-sprite-pipeline/scripts/portrait_pipeline.py` — 2×2
  portrait-loop sheets.
- `<your-output-dir>/generate.py` — `post_process_spritesheet` (the demo's
  spritesheet path, all action-loop assets).
- Any new pipeline that calls `sprite_process.slice_grid_cells` instead of
  rolling its own resize-then-slice loop.

Before adding a new consumer, route it through `slice_grid_cells`. Do not
reintroduce `sheet.resize((cols*cell, rows*cell), …)` followed by a
`cell`-pitch slicer — that is the bug.

## Reference loading hint

Load this file when:
- Spritesheet Phase D (frame extraction) is the active phase
- The pipeline emits a `FrameCountMismatchError`
- Tuning `--chroma-threshold` or `--min-pixels` parameters
- Building a new consumer that slices a raw sheet into grid cells (use
  `sprite_process.slice_grid_cells`)
