Video Generation

JSON-driven video generation pipeline using Remotion. Author timelines as structured JSON, render to MP4 with theme support, TTS narration, transitions, and a full component library — all from the command line.

Default Active: Yes AI & Media

Quick Reference

CLI Commands

All commands are run from the skills_ref/video/ directory. Output goes to /tmp/remotion-render/ by default — keep render output outside the source tree.

Pipeline Usage

# Sync (default) — blocks until render complete
node skills_ref/video/scripts/pipeline.mjs \
  --timeline timeline.json \
  --output /tmp/remotion-render

# With preset override (timeline.meta.preset is the source of truth)
node skills_ref/video/scripts/pipeline.mjs \
  --timeline timeline.json \
  --preset Portrait-1080p

# Async — returns immediately, renders in background
node skills_ref/video/scripts/pipeline.mjs \
  --timeline timeline.json \
  --async

# Check async status
node skills_ref/video/scripts/pipeline.mjs \
  --status /tmp/remotion-render/render-result.json

Timeline Authoring

Videos are defined as JSON timelines with a meta block (title, preset, fps, duration) and an elements array of typed slides. Each element specifies its type, timing, props, and optional transition.

Minimal Example

{
  "meta": {
    "title": "My Video",
    "preset": "Landscape-1080p",
    "fps": 30,
    "totalDurationSec": 15
  },
  "elements": [
    {
      "type": "title",
      "startSec": 0,
      "durationSec": 5,
      "props": { "title": "Hello World", "subtitle": "A demo" },
      "transition": { "type": "fade" }
    },
    {
      "type": "content",
      "startSec": 5,
      "durationSec": 5,
      "props": {
        "header": "Key Points",
        "bulletPoints": ["Fast", "Safe", "Beautiful"]
      },
      "transition": { "type": "slide", "direction": "from-right" }
    },
    {
      "type": "code",
      "startSec": 10,
      "durationSec": 5,
      "props": {
        "code": "const x = 42;",
        "language": "typescript",
        "title": "Example"
      },
      "transition": { "type": "fade" }
    }
  ],
  "audio": []
}

Theme System

Each video should define a unique aesthetic via meta.theme. Fonts are loaded through @remotion/google-fonts for cross-platform rendering. The default theme uses Chakra Petch (display), Outfit (body), and JetBrains Mono (code) with a dark blue background and cyan accent.

{
  "meta": {
    "theme": {
      "aesthetic": "brutalist tech",
      "font": { "display": "Chakra Petch", "body": "Outfit" },
      "color": { "accent": "#FF6B35", "bg": "#0A0A0A" },
      "gradient": {
        "hero": "radial-gradient(circle at 20% 30%, rgba(255,107,53,0.2) 0%, transparent 60%)"
      }
    }
  }
}

Content Design Rules

Do

• Use concise headers without emoji
• Write bullet points as short phrases (max 8 words), 3–4 per slide
• Vary slide types: title, content, code, content — avoid monotony
• Mix transitions: fade, slide, wipe (avoid repeating the same type 3+ times)
• Show real code on code slides, not pseudocode
• Pick a theme aesthetic and commit to it throughout

Avoid

• Emoji in slide titles or headers
• More than 5 bullets on a single slide
• Generic headers like "Key Features", "Getting Started", or "Summary"
• Inter, Roboto, or Arial as fonts in meta.theme

Content Density

The canvas is large (1920x1080 or 1080x1920). Sparse content creates dead space. Add more content blocks — not bigger fonts or more padding. Each slide should use at least 70% of the canvas area.

Shorts (Portrait-1080p)

• Max 8–10 elements for 60 seconds
• First slide = hook (max 5 words) + descriptive subtitle
• Last slide = CTA or memorable closing + tagline
• 5–6 bullets per content slide, 6–10 lines on code slides
• Use content + bulletPoints together on every content slide

Resolution Presets

Default: Landscape-1080p. The agent auto-selects based on keywords — "reels", "shorts", or "TikTok" triggers Portrait-1080p; "Instagram" triggers Square-1080p.

Component Library

The Remotion project ships 11 slide components plus supporting features. Each maps to a type value in the timeline JSON.

Slide Components

Features

TTS Integration

Three TTS providers are available. The default is Gemini. The pipeline auto-detects narration fields in timeline elements and generates per-cut audio files.

Draft-to-Final Workflow

1. Write timeline.draft.json with narration and optional voiceControl per element.
2. Pipeline auto-detects narration fields and generates per-cut audio.
3. Produces timeline.final.json with audio[] entries and corrected durationSec.
4. Final timeline is rendered with synchronized audio.

Draft Timeline with Narration

{
  "elements": [
    {
      "id": "intro",
      "type": "title",
      "durationSec": 11,
      "narration": "Welcome to the analysis report.",
      "voiceControl": {
        "tonePrompt": "Calm, professional news anchor tone"
      },
      "props": {
        "title": "Tech Trends",
        "subtitle": "2026 Edition"
      }
    }
  ]
}

Duration Estimation

• Korean: ~6.5 chars/sec — Math.ceil(narration.length / 6.5) + 0.5
• Final durationSec is auto-corrected by ffprobe measurement after TTS generation

Render Validation — 3-Tier Gate

All three gates must pass for a render to be considered successful.

node scripts/validate-artifact.mjs /tmp/remotion-render/TimelineVideo.mp4 --preset Landscape-1080p

FFmpeg Patterns

FFmpeg handles deterministic cuts, batch processing, and preprocessing outside of Remotion. These patterns are available for use within the video pipeline.

Extract segment by timestamp

ffmpeg -i raw.mp4 -ss 00:12:30 -to 00:15:45 -c copy segment_01.mp4

Batch cut from edit decision list

#!/bin/bash
# cuts.txt format: start,end,label
while IFS=, read -r start end label; do
  ffmpeg -i raw.mp4 -ss "$start" -to "$end" -c copy "segments/${label}.mp4"
done < cuts.txt

Concatenate segments

for f in segments/*.mp4; do echo "file '$f'"; done > concat.txt
ffmpeg -f concat -safe 0 -i concat.txt -c copy assembled.mp4

Create proxy for faster editing

ffmpeg -i raw.mp4 -vf "scale=960:-2" -c:v libx264 -preset ultrafast -crf 28 proxy.mp4

Extract audio for transcription

ffmpeg -i raw.mp4 -vn -acodec pcm_s16le -ar 16000 audio.wav

Normalize audio levels

ffmpeg -i segment.mp4 -af loudnorm=I=-16:TP=-1.5:LRA=11 -c:v copy normalized.mp4

Scene detection

# Detect scene changes (threshold 0.3 = moderate sensitivity)
ffmpeg -i input.mp4 -vf "select='gt(scene,0.3)',showinfo" -vsync vfr -f null - 2>&1 | grep showinfo

Social media reframing

# 16:9 to 9:16 (center crop for TikTok/Reels)
ffmpeg -i input.mp4 -vf "crop=ih*9/16:ih,scale=1080:1920" vertical.mp4

# 16:9 to 1:1 (center crop for Instagram)
ffmpeg -i input.mp4 -vf "crop=ih:ih,scale=1080:1080" square.mp4

ElevenLabs Voice (Supplementary)

For professional voiceover outside the Remotion TTS pipeline, ElevenLabs is available via direct API. Use when you need high-fidelity English narration, voice cloning, or emotion control beyond what Gemini/Supertone provide. Requires ELEVENLABS_API_KEY.

import os, requests

resp = requests.post(
    f"https://api.elevenlabs.io/v1/text-to-speech/<voice_id>",
    headers={
        "xi-api-key": os.environ["ELEVENLABS_API_KEY"],
        "Content-Type": "application/json"
    },
    json={
        "text": "Your narration text here",
        "model_id": "eleven_turbo_v2_5",
        "voice_settings": {"stability": 0.5, "similarity_boost": 0.75}
    }
)
with open("voiceover.mp3", "wb") as f:
    f.write(resp.content)

Final Polish (Descript / CapCut)

For tasks that code-driven rendering handles poorly, use a traditional editor as the last mile:

• Pacing — adjust cuts that feel too fast or slow
• Captions — auto-generated, then manually cleaned
• Color grading — basic correction and mood
• Final audio mix — balance voice, music, and SFX levels
• Export — platform-specific formats and quality settings

Dependencies

The video pipeline requires Node.js, pnpm, FFmpeg, and a Chromium binary (auto-installed by Remotion).

Environment Variables

Bootstrap

# First-time setup: installs packages and ensures Chromium is available
node scripts/ensure-remotion.mjs

# Verify setup
cd remotion-project && pnpm exec remotion studio

Project Structure

skills_ref/video/
+-- SKILL.md
+-- reference/                     # visual-quality.md, tts-integration.md, components.md
+-- scripts/
|   +-- pipeline.mjs               # CLI entrypoint (sync/async/TTS)
|   +-- tts.mjs                    # TTS orchestrator (multi-provider)
|   +-- tts-providers/             # gemini.mjs, supertone.mjs, supertonic.mjs
|   +-- ensure-remotion.mjs        # Runtime bootstrap
|   +-- validate-artifact.mjs      # Gate 3: ffprobe validation
|   +-- presets.mjs                # Resolution presets (ESM)
+-- remotion-project/
    +-- public/example-timeline.json
    +-- src/
        +-- components/            # 11 slide components + barrel
        +-- timeline/              # JSON-to-React engine

"~해줌" Usage Examples

Real-world examples of how to invoke video tasks in natural language. Korean and English both work.

Troubleshooting

Remotion Chromium not found

If remotion render fails with a browser error, ensure Chromium is installed:

node scripts/ensure-remotion.mjs
# Or manually:
cd remotion-project && pnpm exec remotion browser ensure

FFmpeg not found

Install FFmpeg and ffprobe:

# macOS
brew install ffmpeg

# Ubuntu/Debian
sudo apt-get install -y ffmpeg

# Verify
ffmpeg -version && ffprobe -version

TTS fails with auth error

Check that the appropriate environment variable is set for your chosen provider:

# Gemini (default)
echo $GEMINI_API_KEY

# Supertone
echo $SUPERTONE_API_KEY

# Supertonic requires no key (runs offline)

Render output is empty or corrupt

Run the artifact validator to diagnose:

node scripts/validate-artifact.mjs /tmp/remotion-render/TimelineVideo.mp4 --preset Landscape-1080p

If Gate 3 fails, check the Remotion logs for rendering errors. Common causes: missing fonts, invalid timeline JSON, or out-of-memory on large compositions.

Duration mismatch after TTS

The pipeline auto-corrects durationSec via ffprobe measurement after TTS generation. If durations still seem off, verify the narration text length matches the estimation formula: Math.ceil(text.length / 6.5) + 0.5 for Korean text.