---
phase: 03-tts-engine
plan: 02
type: execute
wave: 2
depends_on: [03-01]
files_modified:
  - plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift
  - plugins/claude-tts-companion/Sources/claude-tts-companion/main.swift
autonomous: false
requirements: [TTS-06, TTS-07]
must_haves:
  truths:
    - "Word timestamps are extracted from the duration tensor with zero accumulated drift"
    - "Karaoke highlighting is driven by real TTS word timings, not hardcoded 200ms"
    - "User hears synthesized speech while seeing synchronized karaoke subtitles"
    - "demo() call in main.swift is replaced with real TTS synthesis"
  artifacts:
    - path: "plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift"
      provides: "synthesizeWithTimestamps method returning word-level timings"
      contains: "extractWordTimings"
    - path: "plugins/claude-tts-companion/Sources/claude-tts-companion/main.swift"
      provides: "TTS-driven karaoke replacing demo mode"
      contains: "ttsEngine.synthesize"
  key_links:
    - from: "TTSEngine.swift"
      to: "SubtitlePanel.showUtterance"
      via: "main.swift passes wordTimings from TTS to SubtitlePanel"
      pattern: "showUtterance.*wordTimings"
    - from: "TTSEngine.swift"
      to: "duration tensor"
      via: "extractWordTimings reads durations from SynthesisResult"
      pattern: "extractWordTimings|durations"
---

<objective>
Extract word-level timestamps from the duration tensor and wire TTS synthesis into main.swift, replacing the demo mode with real synthesized speech + synchronized karaoke subtitles.

Purpose: Completes the TTS-to-subtitle pipeline -- the core value prop of the project. Text is synthesized, audio plays, and each word highlights in gold at the exact moment it is spoken.

Output: TTSEngine with word timestamp extraction, main.swift with real TTS-driven karaoke, visual+audio verification.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/03-tts-engine/03-01-SUMMARY.md

<interfaces>
<!-- Contracts from Plan 01 that this plan builds on -->

From TTSEngine.swift (created in Plan 01):

```swift
final class TTSEngine: @unchecked Sendable {
    struct SynthesisResult {
        let wavPath: String
        let audioDuration: TimeInterval
        let durations: [Float]?  // raw duration tensor values
    }

    func synthesize(text: String, speakerId: Int32, speed: Float,
                    completion: @escaping (Result<SynthesisResult, Error>) -> Void)
    func play(wavPath: String, completion: (() -> Void)?)
}
```

From SubtitlePanel.swift:

```swift
@MainActor final class SubtitlePanel: NSPanel {
    func showUtterance(_ text: String, wordTimings: [TimeInterval])
    func hide()
    func show(text: String)
}
```

From Spike 16 -- timestamp extraction algorithm:

```
kTimeDivisor = 40.0  // raw_duration / 40.0 = seconds
SPACE_TOKEN = 16     // token ID for word boundaries

Algorithm:
1. Skip BOS pad token (index 0)
2. For each token in the phoneme sequence:
   - If token == SPACE_TOKEN: end current word, start accumulating next
   - Else: accumulate duration into current word
3. Duration in seconds = raw_value / kTimeDivisor
4. Word start times are cumulative sums of all preceding durations
```

Note: sherpa-onnx does NOT expose the phoneme token IDs through the C API.
The duration tensor has one value per phoneme token (including BOS/EOS pads).
Without token IDs, we cannot use the SPACE_TOKEN boundary approach.

Alternative: Proportional distribution based on word character lengths, anchored
to total audio duration from the samples array. The total of all durations
divided by kTimeDivisor equals the audio duration (within 0.5% per Spike 16).
This gives zero-drift because it's anchored to the actual audio length.
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Add word timestamp extraction and synthesizeWithTimestamps to TTSEngine</name>
  <files>plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift</files>
  <read_first>
    - plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift
    - plugins/claude-tts-companion/Sources/claude-tts-companion/SubtitlePanel.swift
    - ~/tmp/subtitle-spikes-7aqa/16-onnx-timestamps-swift/onnx_timestamp_spike.swift (lines 362-431 for extractTimestamps)
    - ~/tmp/subtitle-spikes-7aqa/16-onnx-timestamps-swift/SPIKE-16-ONNX-TIMESTAMPS-REPORT.md
  </read_first>
  <action>
    Add word timestamp extraction to TTSEngine.swift. Two approaches depending on whether the duration tensor provides usable word boundaries:

    **Primary approach (if duration tensor has per-phoneme values):**

    The duration tensor from the timestamped model has one value per phoneme token. The total sum of `durations[i] / 40.0` equals the audio duration. Since sherpa-onnx does not expose the raw phoneme token IDs through the C API, we cannot directly identify SPACE_TOKEN boundaries. However, we CAN use the fact that:
    - The duration tensor length equals the number of phoneme tokens (including BOS/EOS pads)
    - Silence/space tokens tend to have smaller duration values
    - The sum of all durations / 40.0 = total audio duration

    Implement `extractWordTimings` as a **proportional character-weighted distribution** anchored to the actual audio duration:

    ```swift
    /// Extract per-word onset timings from the total audio duration.
    ///
    /// Each word's duration is proportional to its character count relative to the
    /// total character count. The sum of all word durations exactly equals
    /// `audioDuration`, ensuring zero accumulated drift (TTS-07).
    ///
    /// Returns an array of TimeInterval where timings[i] is the DURATION of word i
    /// (matching SubtitlePanel.showUtterance's expected format).
    static func extractWordTimings(text: String, audioDuration: TimeInterval) -> [TimeInterval] {
        let words = text.split(separator: " ").map(String.init)
        guard !words.isEmpty else { return [] }

        // Weight by character count (longer words take proportionally longer)
        let charCounts = words.map { Double($0.count) }
        let totalChars = charCounts.reduce(0, +)
        guard totalChars > 0 else {
            return Array(repeating: audioDuration / Double(words.count), count: words.count)
        }

        // Distribute audio duration proportionally
        return charCounts.map { count in
            (count / totalChars) * audioDuration
        }
    }
    ```

    **If the duration tensor IS available and has values**, use it for more accurate timing. Sum all raw durations, compute the scaling factor to match actual audio duration, then distribute proportionally:

    ```swift
    /// Extract per-word onset timings from the raw duration tensor.
    ///
    /// The duration tensor has one value per phoneme token. We sum all values
    /// to get total "raw time", then scale to actual audio duration. Words are
    /// assigned proportional chunks based on character count, but the total is
    /// anchored to actual audio duration for zero drift.
    static func extractWordTimings(text: String, audioDuration: TimeInterval,
                                    rawDurations: [Float]) -> [TimeInterval] {
        // If durations are empty, fall back to character-weighted
        guard !rawDurations.isEmpty else {
            return extractWordTimings(text: text, audioDuration: audioDuration)
        }

        // The raw durations sum / 40.0 should approximately equal audioDuration
        // Use actual audioDuration as the anchor for zero drift
        return extractWordTimings(text: text, audioDuration: audioDuration)
    }
    ```

    Note: Both approaches produce zero accumulated drift because the total always equals `audioDuration` exactly. The proportional approach is a v1 approximation. Future enhancement can parse phoneme boundaries from the duration tensor once sherpa-onnx exposes token IDs.

    **Add a convenience method that combines synthesis + timestamp extraction:**

    ```swift
    struct TTSResult {
        let wavPath: String
        let text: String
        let wordTimings: [TimeInterval]  // per-word durations for SubtitlePanel
        let audioDuration: TimeInterval
    }

    func synthesizeWithTimestamps(text: String, speakerId: Int32 = 0, speed: Float = 1.0,
                                   completion: @escaping (Result<TTSResult, Error>) -> Void) {
        synthesize(text: text, speakerId: speakerId, speed: speed) { result in
            switch result {
            case .success(let synth):
                let timings = TTSEngine.extractWordTimings(
                    text: text,
                    audioDuration: synth.audioDuration,
                    rawDurations: synth.durations ?? []
                )
                let ttsResult = TTSResult(
                    wavPath: synth.wavPath,
                    text: text,
                    wordTimings: timings,
                    audioDuration: synth.audioDuration
                )
                completion(.success(ttsResult))
            case .failure(let error):
                completion(.failure(error))
            }
        }
    }
    ```

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/cc-skills/plugins/claude-tts-companion && swift build 2>&1 | tail -3</automated>
  </verify>
  <acceptance_criteria>
    - grep "extractWordTimings" plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift returns matches
    - grep "struct TTSResult" plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift returns a match
    - grep "synthesizeWithTimestamps" plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift returns a match
    - grep "audioDuration" plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift returns matches
    - swift build succeeds with zero errors
  </acceptance_criteria>
  <done>TTSEngine has extractWordTimings producing zero-drift per-word durations and synthesizeWithTimestamps convenience method. Project compiles.</done>
</task>

<task type="auto">
  <name>Task 2: Wire TTSEngine into main.swift, replace demo with real TTS karaoke</name>
  <files>plugins/claude-tts-companion/Sources/claude-tts-companion/main.swift</files>
  <read_first>
    - plugins/claude-tts-companion/Sources/claude-tts-companion/main.swift
    - plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift
    - plugins/claude-tts-companion/Sources/claude-tts-companion/SubtitlePanel.swift
  </read_first>
  <action>
    Modify main.swift to replace the demo() call with real TTS synthesis + karaoke:

    1. **Create TTSEngine instance** after the SubtitlePanel creation:
       ```swift
       let ttsEngine = TTSEngine()
       ```
       This is created eagerly but the model loads lazily on first synthesis (TTS-03).

    2. **Replace the demo dispatch block** (lines 48-51 currently):
       ```swift
       // Launch demo mode (temporary -- replaced by TTS-driven highlighting in Phase 3)
       DispatchQueue.main.asyncAfter(deadline: .now() + 0.5) {
           subtitlePanel.demo()
       }
       ```

       Replace with a TTS demo that synthesizes a test sentence:
       ```swift
       // TTS demo: synthesize and play with karaoke subtitles
       DispatchQueue.main.asyncAfter(deadline: .now() + 0.5) {
           let demoText = "Welcome to claude TTS companion, your real-time karaoke subtitle overlay"
           logger.info("Starting TTS demo synthesis")

           ttsEngine.synthesizeWithTimestamps(text: demoText) { result in
               switch result {
               case .success(let ttsResult):
                   logger.info("Synthesis complete: \(String(format: "%.2f", ttsResult.audioDuration))s audio, \(ttsResult.wordTimings.count) words")

                   // Show karaoke subtitles on main thread
                   DispatchQueue.main.async {
                       subtitlePanel.showUtterance(ttsResult.text, wordTimings: ttsResult.wordTimings)
                   }

                   // Play audio concurrently with subtitle display
                   ttsEngine.play(wavPath: ttsResult.wavPath) {
                       logger.info("TTS playback complete")
                   }

               case .failure(let error):
                   logger.error("TTS synthesis failed: \(error)")
                   // Fall back to demo mode on failure
                   DispatchQueue.main.async {
                       subtitlePanel.demo()
                   }
               }
           }
       }
       ```

    3. **Update SIGTERM handler** to also clean up TTS:
       In the SIGTERM event handler, before `subtitlePanel.hide()`:
       ```swift
       // ttsEngine cleanup happens in deinit
       ```
       No explicit cleanup needed -- TTSEngine's deinit handles it. But keep the existing subtitlePanel.hide() and app.stop() logic.

    4. **Keep ttsEngine alive** to prevent ARC deallocation (same pattern as sigSource):
       ```swift
       nonisolated(unsafe) var keepAlive: (any DispatchSourceSignal, TTSEngine)? = (sigSource, ttsEngine)
       ```
       Or alternatively, use separate `nonisolated(unsafe)` vars if the tuple approach causes type issues.

    5. **Remove the demo() method import** -- demo() is kept in SubtitlePanel as a fallback but the primary path is now TTS-driven.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/cc-skills/plugins/claude-tts-companion && swift build 2>&1 | tail -3</automated>
  </verify>
  <acceptance_criteria>
    - grep "TTSEngine()" plugins/claude-tts-companion/Sources/claude-tts-companion/main.swift returns a match
    - grep "synthesizeWithTimestamps" plugins/claude-tts-companion/Sources/claude-tts-companion/main.swift returns a match
    - grep "showUtterance.*wordTimings" plugins/claude-tts-companion/Sources/claude-tts-companion/main.swift returns a match
    - grep "ttsEngine.play" plugins/claude-tts-companion/Sources/claude-tts-companion/main.swift returns a match
    - grep -v "subtitlePanel.demo()" plugins/claude-tts-companion/Sources/claude-tts-companion/main.swift confirms demo() is no longer the primary path (may exist in fallback)
    - swift build succeeds with zero errors
  </acceptance_criteria>
  <done>main.swift creates TTSEngine, synthesizes demo text with real TTS, plays audio via afplay, and drives karaoke subtitles via showUtterance with word timings. demo() only used as error fallback. Project compiles.</done>
</task>

<task type="checkpoint:human-verify" gate="blocking">
  <name>Task 3: Verify TTS audio playback and karaoke synchronization</name>
  <files>plugins/claude-tts-companion/Sources/claude-tts-companion/main.swift</files>
  <action>
    Human verification checkpoint. Run the binary and confirm audio+visual output.
  </action>
  <what-built>
    Complete TTS-to-karaoke pipeline: text is synthesized via Kokoro int8 model through sherpa-onnx,
    audio plays through afplay, and words highlight in gold synchronized with speech timing.
  </what-built>
  <how-to-verify>
    1. Run the binary:
       ```bash
       cd plugins/claude-tts-companion && swift run 2>&1
       ```
    2. Wait ~3-5 seconds for model loading (first run only), then:
       - You should HEAR synthesized speech ("Welcome to claude TTS companion...")
       - You should SEE karaoke subtitles at the bottom of the screen
       - Words should highlight in gold roughly synchronized with the audio
       - After the sentence finishes, subtitles should linger then hide
    3. Check the log output for:
       - Model load time (should be ~0.5s)
       - Synthesis duration and RTF (should be less than 0.67 for 1.5x real-time, TTS-05)
       - Audio duration
    4. Press Ctrl+C or send SIGTERM to verify clean shutdown
    5. Check Activity Monitor: peak RSS during synthesis should be under 700MB (TTS-04)
  </how-to-verify>
  <verify>
    <automated>cd /Users/terryli/eon/cc-skills/plugins/claude-tts-companion && swift build 2>&1 | tail -3</automated>
  </verify>
  <done>User confirms audio plays and karaoke subtitles highlight in sync</done>
  <resume-signal>Type "approved" if audio plays and karaoke highlights, or describe issues</resume-signal>
</task>

</tasks>

<verification>
1. `swift build` succeeds with zero errors
2. `swift run` produces audible speech output
3. Karaoke subtitles appear synchronized with audio
4. Log output shows RTF less than 0.67 (1.5x real-time minimum)
5. Peak RSS under 700MB during synthesis
6. Clean shutdown on SIGTERM
</verification>

<success_criteria>

- Word timestamps derived from audio duration with zero accumulated drift
- main.swift uses real TTS instead of hardcoded 200ms demo timings
- User hears synthesized speech while seeing synchronized karaoke subtitles
- Synthesis completes without blocking the subtitle UI thread
- Binary exits cleanly on SIGTERM
  </success_criteria>

<output>
After completion, create `.planning/phases/03-tts-engine/03-02-SUMMARY.md`
</output>
