---
phase: 26-swift-ttsengine-python-integration
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - plugins/claude-tts-companion/Sources/CompanionCore/TTSEngine.swift
autonomous: false
requirements: [SWI-01, SWI-02, SWI-03]

must_haves:
  truths:
    - "TTSEngine calls /v1/audio/speech-with-timestamps instead of /v1/audio/speech for English synthesis"
    - "ChunkResult.wordOnsets is populated with native onset times from Python server JSON response"
    - "TTSResult.wordOnsets is populated with native onset times from Python server JSON response"
    - "SubtitleSyncDriver receives native onsets and uses them for karaoke highlighting (no character-weighted fallback for English)"
    - "tts_kokoro.sh works end-to-end via Swift companion -> Python server chain"
  artifacts:
    - path: "plugins/claude-tts-companion/Sources/CompanionCore/TTSEngine.swift"
      provides: "Python server JSON parsing with word onset extraction"
      contains: "speech-with-timestamps"
  key_links:
    - from: "TTSEngine.callPythonServerWithTimestamps()"
      to: "http://127.0.0.1:8779/v1/audio/speech-with-timestamps"
      via: "URLSession POST with JSON body"
      pattern: "speech-with-timestamps"
    - from: "TTSEngine.synthesizeStreaming()"
      to: "ChunkResult.wordOnsets"
      via: "JSON parsing of words array"
      pattern: "wordOnsets.*onset"
    - from: "TTSPipelineCoordinator.startBatchPipeline()"
      to: "SubtitleSyncDriver.addChunk(nativeOnsets:)"
      via: "chunk.wordOnsets passthrough"
      pattern: "nativeOnsets.*chunk\\.wordOnsets"
---

<objective>
Switch TTSEngine from calling /v1/audio/speech (raw WAV) to /v1/audio/speech-with-timestamps (JSON with WAV + per-word timing). Parse the JSON response to extract native word onsets from the Python MLX server and pass them through the pipeline to SubtitleSyncDriver for zero-drift karaoke highlighting.

Purpose: Eliminate character-weighted timing fallback for English TTS. Native word onsets from the Kokoro duration model produce zero-drift karaoke highlighting vs the visible drift of character-weighted approximation.

Output: TTSEngine.swift updated to parse timestamp JSON, all synthesis paths populate wordOnsets, karaoke subtitles use native timing.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/25-python-tts-server-timestamp-endpoint/25-01-SUMMARY.md

<interfaces>
<!-- Key types and contracts the executor needs. -->

Python server response format (from Phase 25):

```json
{
  "audio_b64": "<base64-encoded WAV bytes>",
  "words": [{"text": "Hello", "onset": 0.0, "duration": 0.45}, ...],
  "audio_duration": 5.67,
  "sample_rate": 24000
}
```

From TTSEngine.swift:

```swift
public struct TTSResult: Sendable {
    let wavPath: String
    let text: String
    let wordTimings: [TimeInterval]
    let audioDuration: TimeInterval
    let wordOnsets: [TimeInterval]?  // Currently always nil -- needs population
}

struct ChunkResult: Sendable {
    let wavPath: String
    let text: String
    let wordTimings: [TimeInterval]
    let audioDuration: TimeInterval
    let chunkIndex: Int
    let totalChunks: Int
    let wordOnsets: [TimeInterval]?  // Currently always nil -- needs population
    let samples: [Float]?
}
```

From SubtitleSyncDriver.swift:

```swift
func addChunk(wavPath: String, samples: [Float]? = nil, pages: [SubtitlePage],
              wordTimings: [TimeInterval], nativeOnsets: [TimeInterval]? = nil)
// Already handles nativeOnsets -- just needs non-nil values passed in
```

Current callPythonServer() calls: /v1/audio/speech -> returns raw WAV Data
Need: /v1/audio/speech-with-timestamps -> returns JSON with audio_b64 + words array

Config.pythonTTSServerURL = "<http://127.0.0.1:8779>" (or KOKORO_TTS_SERVER_URL env)
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Add callPythonServerWithTimestamps and wire all synthesis paths</name>
  <read_first>
    - plugins/claude-tts-companion/Sources/CompanionCore/TTSEngine.swift
    - plugins/claude-tts-companion/Sources/CompanionCore/WordTimingAligner.swift
  </read_first>
  <files>plugins/claude-tts-companion/Sources/CompanionCore/TTSEngine.swift</files>
  <action>
  Add a new private method and response struct to TTSEngine:

1. Add a Codable struct `PythonTimestampResponse` at file scope (inside the actor or as a private struct):

   ```swift
   private struct PythonTimestampWord: Codable {
       let text: String
       let onset: Double
       let duration: Double
   }
   private struct PythonTimestampResponse: Codable {
       let audio_b64: String
       let words: [PythonTimestampWord]
       let audio_duration: Double
       let sample_rate: Int
   }
   ```

2. Add a new private method `callPythonServerWithTimestamps(text:voice:speed:)` that:
   - POSTs to `Config.pythonTTSServerURL + "/v1/audio/speech-with-timestamps"` (not `/v1/audio/speech`)
   - Sends the same JSON body as callPythonServer (input, voice, language, speed, response_format: "wav")
   - Parses the JSON response into PythonTimestampResponse
   - Decodes audio_b64 from base64 into Data (WAV bytes)
   - Returns a tuple: `(wavData: Data, wordOnsets: [TimeInterval], wordDurations: [TimeInterval], audioDuration: TimeInterval)`
   - On base64 decode failure, throw TTSError.synthesisReturnedNil

3. Update `synthesize()` to call `callPythonServerWithTimestamps` instead of `callPythonServer`. It only needs the WAV data and audioDuration (discard word timing for this path).

4. Update `synthesizeWithTimestamps()` to:
   - Call `callPythonServerWithTimestamps` instead of `callPythonServer`
   - Use the returned wordDurations for `wordTimings` (these are native per-word durations, not character-weighted)
   - Use the returned wordOnsets for `wordOnsets` (pass through to TTSResult)
   - Remove the `WordTimingAligner.extractWordTimings()` fallback call for English path
   - Still keep wordTimings populated (SubtitleSyncDriver uses them as fallback if onset count mismatches)

5. Update `synthesizeStreaming()` to:
   - In the per-sentence synthesis loop, call `callPythonServerWithTimestamps` instead of `callPythonServer`
   - Extract wordOnsets and wordDurations from the response
   - Pass wordDurations as `wordTimings` in ChunkResult (instead of character-weighted)
   - Pass wordOnsets as `wordOnsets` in ChunkResult (instead of nil)
   - Remove the `WordTimingAligner.extractWordTimings()` call inside the loop

6. Keep `callPythonServer()` as-is for now (it may be needed as fallback or can be removed in a later cleanup). Just stop calling it from the main paths.

Do NOT modify WordTimingAligner.swift in this task -- it still has MToken-based methods that will be cleaned up in Phase 27 when MLX is removed entirely.
</action>
<acceptance_criteria> - grep -c "speech-with-timestamps" plugins/claude-tts-companion/Sources/CompanionCore/TTSEngine.swift returns >= 2 (URL string + method/comment) - grep -c "PythonTimestampResponse" plugins/claude-tts-companion/Sources/CompanionCore/TTSEngine.swift returns >= 2 - grep -c "audio_b64" plugins/claude-tts-companion/Sources/CompanionCore/TTSEngine.swift returns >= 1 - grep "wordOnsets: nil" plugins/claude-tts-companion/Sources/CompanionCore/TTSEngine.swift should return 0 matches for English paths (only CJK path should have nil)
</acceptance_criteria>
<verify>
<automated>cd /Users/terryli/eon/cc-skills/plugins/claude-tts-companion && swift build 2>&1 | tail -5</automated>
</verify>
<done> - callPythonServerWithTimestamps() calls /v1/audio/speech-with-timestamps and parses JSON response - synthesize(), synthesizeWithTimestamps(), and synthesizeStreaming() all use the new method - ChunkResult.wordOnsets and TTSResult.wordOnsets populated with native onset times from Python server - Character-weighted WordTimingAligner.extractWordTimings() no longer called for English synthesis paths - swift build succeeds with zero errors
</done>
</task>

<task type="checkpoint:human-verify" gate="blocking">
  <name>Task 2: Verify karaoke highlighting uses native word onsets end-to-end</name>
  <files>plugins/claude-tts-companion/Sources/CompanionCore/TTSEngine.swift</files>
  <action>
    Human verifies that karaoke subtitle highlighting is precisely synced with TTS audio playback using native word onsets from the Python server. This replaces the character-weighted fallback timing that caused visible drift on multi-syllable words.
  </action>
  <what-built>
    TTSEngine now delegates to Python server's /v1/audio/speech-with-timestamps endpoint and passes native word onset times through to SubtitleSyncDriver for zero-drift karaoke highlighting. The tts_kokoro.sh CLI script works through the same chain.
  </what-built>
  <how-to-verify>
    1. Ensure both services are running:
       - `curl -s http://127.0.0.1:8779/health` (Python TTS server)
       - `curl -s http://127.0.0.1:8780/health` (Swift companion)
    2. Test via tts_kokoro.sh (SWI-03):
       ```bash
       ~/.local/bin/tts_kokoro.sh "The quick brown fox jumps over the lazy dog near the riverbank."
       ```
    3. Watch the subtitle overlay -- gold karaoke highlighting should advance word-by-word in sync with audio playback
    4. Test a longer passage to verify zero drift (SWI-02):
       ```bash
       ~/.local/bin/tts_kokoro.sh "Claude is an AI assistant made by Anthropic. It helps people with coding, writing, analysis, and many other tasks. The karaoke subtitles should highlight each word precisely as it is spoken, with no accumulated timing drift even across multiple sentences."
       ```
    5. Verify highlighting stays precisely synced through the entire passage (no word highlighted too early or too late)
  </how-to-verify>
  <verify>
    <automated>cd /Users/terryli/eon/cc-skills/plugins/claude-tts-companion && swift build 2>&1 | tail -3</automated>
  </verify>
  <done>Karaoke highlighting precisely synced with audio -- no visible drift on multi-syllable words or across sentence boundaries</done>
  <resume-signal>Type "approved" if karaoke highlighting is precisely synced, or describe any timing drift observed</resume-signal>
</task>

</tasks>

<verification>
1. `swift build` succeeds in plugins/claude-tts-companion/
2. TTSEngine.swift contains "speech-with-timestamps" endpoint URL
3. TTSEngine.swift contains PythonTimestampResponse JSON parsing
4. No character-weighted fallback called for English synthesis paths
5. tts_kokoro.sh produces audio with synced karaoke subtitles
</verification>

<success_criteria>

- TTSEngine sends text to Python server via /v1/audio/speech-with-timestamps HTTP endpoint (SWI-01)
- Native word onsets from Python response drive karaoke highlighting with zero accumulated drift (SWI-02)
- tts_kokoro.sh works end-to-end via Swift companion -> Python server chain (SWI-03)
  </success_criteria>

<output>
After completion, create `.planning/phases/26-swift-ttsengine-python-integration/26-01-SUMMARY.md`
</output>
