---
phase: quick
plan: 260327-d2e
type: execute
wave: 1
depends_on: []
files_modified:
  - plugins/claude-tts-companion/Package.swift
  - plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift
  - plugins/claude-tts-companion/Sources/claude-tts-companion/Config.swift
  - plugins/claude-tts-companion/Sources/claude-tts-companion/LanguageDetector.swift
  - plugins/claude-tts-companion/Sources/claude-tts-companion/TelegramBot.swift
autonomous: true
must_haves:
  truths:
    - "TTS synthesis produces audible speech using kokoro-ios MLX Metal GPU"
    - "Word timestamps from TimestampPredictor drive karaoke highlighting"
    - "Streaming sentence-chunked synthesis pipeline works with new engine"
    - "Binary builds successfully with swift build -c release"
  artifacts:
    - path: "plugins/claude-tts-companion/Package.swift"
      provides: "kokoro-ios + mlx-swift SPM deps, no sherpa-onnx"
      contains: "kokoro-ios"
    - path: "plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift"
      provides: "KokoroTTS-based synthesis with native word timestamps"
      contains: "KokoroTTS"
    - path: "plugins/claude-tts-companion/Sources/claude-tts-companion/Config.swift"
      provides: "MLX model paths and kokoro-ios configuration"
      contains: "safetensors"
  key_links:
    - from: "TTSEngine.swift"
      to: "KokoroSwift"
      via: "import KokoroSwift; KokoroTTS.generateAudio()"
      pattern: "tts\\.generateAudio"
    - from: "TTSEngine.swift"
      to: "SubtitleSyncDriver"
      via: "MToken.start_ts/end_ts -> wordTimings array"
      pattern: "start_ts.*end_ts"
---

<objective>
Replace sherpa-onnx CPU TTS backend with kokoro-ios MLX Metal GPU engine.

Purpose: 1.5-2x faster synthesis (RTF 0.096 vs 0.15-0.20), native word timestamps (no C++ patches), pure Swift stack (no C interop/static libs).
Output: Working TTS pipeline using kokoro-ios with karaoke subtitle sync.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@plugins/claude-tts-companion/Package.swift
@plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift
@plugins/claude-tts-companion/Sources/claude-tts-companion/Config.swift
@plugins/claude-tts-companion/Sources/claude-tts-companion/SubtitleSyncDriver.swift
@plugins/claude-tts-companion/Sources/claude-tts-companion/TelegramBot.swift
@plugins/claude-tts-companion/Sources/claude-tts-companion/LanguageDetector.swift

<interfaces>
<!-- Spike reference at ~/tmp/kokoro-mlx-spike/ — working patterns to port -->

From spike Package.swift:

```swift
.package(url: "https://github.com/mlalma/kokoro-ios.git", from: "1.0.0"),
.package(url: "https://github.com/mlalma/MLXUtilsLibrary.git", from: "0.0.6"),
.package(url: "https://github.com/ml-explore/mlx-swift", from: "0.30.0"),
// Products: "KokoroSwift", "MLXUtilsLibrary", "MLX"
// Platform: .macOS(.v15)
```

From spike main.swift — synthesis API:

```swift
import KokoroSwift
import MLX
import MLXUtilsLibrary

let tts = KokoroTTS(modelPath: modelURL)
let voices = NpyzReader.read(fileFromPath: voicesURL)  // voices.npz
let voice = voices["af_heart"]!                         // MLXArray

let (audio, tokenArray) = try tts.generateAudio(
    voice: voice, language: .enUS, text: text, speed: 1.0
)
// audio: [Float], tokenArray: [MToken]?
// MToken.start_ts: Double?, MToken.end_ts: Double?, MToken.text: String
// Sample rate: 24000 Hz
```

From spike — WAV writing (no SherpaOnnxWriteWave available):

```swift
// Use AVAudioFile to write WAV from [Float] samples
let format = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 24000, channels: 1, interleaved: false)
let buffer = AVAudioPCMBuffer(pcmFormat: format!, frameCapacity: AVAudioFrameCount(samples.count))
buffer!.frameLength = AVAudioFrameCount(samples.count)
let channelData = buffer!.floatChannelData![0]
for i in 0..<samples.count { channelData[i] = samples[i] }
let audioFile = try AVAudioFile(forWriting: url, settings: format!.settings, commonFormat: format!.commonFormat, interleaved: format!.isInterleaved)
try audioFile.write(from: buffer!)
```

Model files location (spike used ~/tmp/kokoro-mlx-spike/models/):

- kokoro-v1_0.safetensors (327 MB, bf16)
- voices.npz (voice embeddings)

Runtime requirement: mlx.metallib (102 MB) must be next to binary or in a known path.

Current TTSEngine public API (must preserve):

- synthesize(text:speakerId:speed:completion:)
- synthesizeWithTimestamps(text:speakerId:speed:completion:)
- synthesizeStreaming(text:speakerId:speed:onChunkReady:onAllComplete:)
- play(wavPath:completion:) -> AVAudioPlayer?
- stopPlayback()

Current types (must preserve):

- SynthesisResult { wavPath, audioDuration, durations }
- TTSResult { wavPath, text, wordTimings, audioDuration }
- ChunkResult { wavPath, text, wordTimings, audioDuration, chunkIndex, totalChunks }
  </interfaces>

</context>

<tasks>

<task type="auto">
  <name>Task 1: Replace Package.swift deps and Config paths</name>
  <files>
    plugins/claude-tts-companion/Package.swift
    plugins/claude-tts-companion/Sources/claude-tts-companion/Config.swift
  </files>
  <action>
**Package.swift changes:**
1. Bump platform from `.macOS(.v14)` to `.macOS(.v15)` (required by kokoro-ios)
2. Remove the `CSherpaOnnx` target entirely
3. Remove the entire `linkerSettings` block from the executable target (no more `-L.../lib`, `-lsherpa-onnx`, `-lonnxruntime`, `-lc++`)
4. Remove `"CSherpaOnnx"` from dependencies of the executable target
5. Remove the `sherpaOnnxPath` variable at the top
6. Add these SPM dependencies:
   ```swift
   .package(url: "https://github.com/mlalma/kokoro-ios.git", from: "1.0.0"),
   .package(url: "https://github.com/mlalma/MLXUtilsLibrary.git", from: "0.0.6"),
   .package(url: "https://github.com/ml-explore/mlx-swift", from: "0.30.0"),
   ```
7. Add to executable target dependencies:
   ```swift
   .product(name: "KokoroSwift", package: "kokoro-ios"),
   .product(name: "MLXUtilsLibrary", package: "MLXUtilsLibrary"),
   .product(name: "MLX", package: "mlx-swift"),
   ```
8. Keep existing deps: swift-telegram-sdk, swift-log, FlyingFox

**Config.swift changes:**

1. Remove `sherpaOnnxPath` property entirely
2. Remove `kokoroModelFile` property (no more .onnx file detection)
3. Rename `kokoroModelPath` to `kokoroMLXModelPath` and change default to:
   `~/.local/share/kokoro/models/mlx/kokoro-v1_0.safetensors`
   with env override `KOKORO_MLX_MODEL_PATH`
4. Add `kokoroVoicesPath`: `~/.local/share/kokoro/models/mlx/voices.npz`
   with env override `KOKORO_VOICES_PATH`
5. Add `kokoroMLXMetallibPath`: path to `mlx.metallib`, default to same dir as the binary
   (use `Bundle.main.executableURL?.deletingLastPathComponent()` or env `MLX_METALLIB_PATH`)
6. Add `defaultVoiceName: String = "af_heart"` (currently hardcoded speaker ID 3)
7. Add `chineseVoiceName: String = "zf_xiaobei"` (for CJK fallback — note: kokoro-ios may not support Chinese; keep the name for future use)
8. Keep `defaultSpeakerId` and `chineseSpeakerId` as deprecated/unused (or remove — they were sherpa-onnx speaker IDs)
9. Add `useSherpaFallback: Bool` reading env `USE_SHERPA_FALLBACK` (default false). This is the escape hatch if kokoro-ios has issues. When true AND CJK text detected, log a warning that Chinese TTS is not supported by kokoro-ios and skip TTS.

**Do NOT delete** the `Sources/CSherpaOnnx/` directory yet — that will be a separate cleanup step after validation.
</action>
<verify>
<automated>cd plugins/claude-tts-companion && swift package resolve 2>&1 | tail -5</automated>
</verify>
<done>Package resolves with kokoro-ios + mlx-swift deps. Config points to MLX model paths. No sherpa-onnx linker flags.</done>
</task>

<task type="auto">
  <name>Task 2: Rewrite TTSEngine for kokoro-ios and update callers</name>
  <files>
    plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift
    plugins/claude-tts-companion/Sources/claude-tts-companion/LanguageDetector.swift
    plugins/claude-tts-companion/Sources/claude-tts-companion/TelegramBot.swift
  </files>
  <action>
**TTSEngine.swift — full rewrite of internals, preserve public API signatures:**

1. Replace imports: remove `CSherpaOnnx`, add `KokoroSwift`, `MLX`, `MLXUtilsLibrary`
2. Replace private state:
   - Remove `ttsInstance: OpaquePointer?`
   - Add `ttsInstance: KokoroTTS?` (lazy loaded)
   - Add `voice: MLXArray?` (loaded from voices.npz)
   - Add `voicesDict: [String: MLXArray]?` (all voices for voice switching)
3. Rewrite `ensureModelLoaded()`:
   - Create `KokoroTTS(modelPath: URL(fileURLWithPath: Config.kokoroMLXModelPath))`
   - Load voices: `NpyzReader.read(fileFromPath: URL(fileURLWithPath: Config.kokoroVoicesPath))`
   - Extract default voice: `voicesDict[Config.defaultVoiceName]`
   - Return the KokoroTTS instance
   - Log model load time and voice count
4. Add private method `voiceForLanguage(_ lang: String) -> MLXArray`:
   - If lang == "cmn" or starts with "zh", log warning "kokoro-ios does not support Chinese, using English voice" and return default voice
   - Otherwise return default voice (af_heart)
   - Future: look up `Config.chineseVoiceName` in voicesDict if kokoro-ios adds Chinese support
5. Rewrite `synthesize()`:
   - Call `ensureModelLoaded()` to get tts instance
   - Determine language from speakerId: if speakerId == Config.chineseSpeakerId, use `.enUS` with warning (kokoro-ios is English-only)
   - Call `let (audio, tokenArray) = try tts.generateAudio(voice: voice, language: .enUS, text: text, speed: speed)`
   - `audio` is `[Float]`, compute `audioDuration = Double(audio.count) / 24000.0`
   - Write WAV using AVAudioFile (port the writeWav helper from spike — AVAudioFormat pcmFormatFloat32, 24kHz mono)
   - Do NOT use `SherpaOnnxWriteWave` (removed)
   - Build `SynthesisResult` — set `durations` to nil (we use MToken timestamps instead)
   - Log RTF as before
6. Rewrite `synthesizeWithTimestamps()`:
   - Similar to synthesize but also process `tokenArray: [MToken]?`
   - Extract word timings from MToken array: for each token with `start_ts` and `end_ts`, compute duration as `end_ts - start_ts`
   - Map MToken timestamps to the `wordTimings: [TimeInterval]` array format expected by SubtitleSyncDriver (array of durations per word)
   - Handle punctuation tokens (comma, period) — include their duration but merge with adjacent word or create zero-width entries
   - Fallback: if tokenArray is nil or empty, use character-weighted distribution (existing `extractWordTimings(text:audioDuration:)`)
7. Rewrite `synthesizeStreaming()`:
   - Same sentence splitting logic (keep `splitIntoSentences`)
   - For each sentence, call `tts.generateAudio(voice:language:text:speed:)`
   - Write WAV using the new AVAudioFile helper
   - Extract word timings from MToken array per chunk
   - Build ChunkResult with native timestamps
8. Add private `writeWav(samples:sampleRate:to:)` method — port from spike:

   ```swift
   private func writeWav(samples: [Float], sampleRate: Double = 24000.0, to path: String) throws {
       let url = URL(fileURLWithPath: path)
       guard let format = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: sampleRate, channels: 1, interleaved: false),
             let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(samples.count)) else {
           throw TTSError.wavWriteFailed(path: path)
       }
       buffer.frameLength = AVAudioFrameCount(samples.count)
       let channelData = buffer.floatChannelData![0]
       for i in 0..<samples.count { channelData[i] = samples[i] }
       let audioFile = try AVAudioFile(forWriting: url, settings: format.settings, commonFormat: format.commonFormat, interleaved: format.isInterleaved)
       try audioFile.write(from: buffer)
   }
   ```

9. Add private helper `extractTimingsFromTokens(_ tokens: [MToken]?) -> [TimeInterval]`:
   - Guard tokens not nil/empty, return [] on fallback
   - Filter to tokens where both start_ts and end_ts are non-nil
   - Skip punctuation-only tokens (where text is just ".", ",", "!", "?", ";", ":", "-")
   - For each word token, duration = end_ts - start_ts
   - Return array of durations
10. Keep `play()`, `stopPlayback()`, `splitIntoSentences()` unchanged
11. Remove old `extractWordTimings(text:audioDuration:rawDurations:)` overload (raw durations was for sherpa-onnx tensor data). Keep the character-weighted `extractWordTimings(text:audioDuration:)` as fallback.
12. Update `deinit` — remove `SherpaOnnxDestroyOfflineTts` call, just nil out ttsInstance

**LanguageDetector.swift — minor update:**

- Change `LanguageResult.speakerId` type from `Int32` to `String` representing voice name
- Return `Config.defaultVoiceName` ("af_heart") for English, `Config.chineseVoiceName` ("zf_xiaobei") for Chinese
- OR: Keep speakerId as Int32 but add a `voiceName: String` field — this is cleaner since TelegramBot uses speakerId in dispatchTTS
- Actually simplest: keep LanguageResult as-is but **add** a `voiceName: String` field. English -> "af_heart", Chinese -> "af_heart" (with warning, since kokoro-ios is English-only)

**TelegramBot.swift — update dispatchTTS callers:**

- In `dispatchTTS()`: pass `langResult.voiceName` to synthesis methods instead of (or in addition to) speakerId
- Update `dispatchStreamingTTS` and `dispatchFullTTS` signatures to accept voiceName: String
- In TTSEngine synthesis methods: accept `voiceName: String` parameter instead of `speakerId: Int32`
  - Look up voice in `voicesDict` by name, fallback to default voice
  - This replaces the integer speaker ID system with string voice names (kokoro-ios uses voice names in NPZ)
- Update the `/health` command in handleHealth: change "Kokoro int8" to "Kokoro MLX (bf16)"
  </action>
  <verify>
  <automated>cd plugins/claude-tts-companion && swift build 2>&1 | tail -20</automated>
  </verify>
  <done>
- `swift build` succeeds with zero errors
- TTSEngine uses KokoroTTS.generateAudio() for synthesis
- Word timestamps extracted from MToken.start_ts/end_ts (not character-weighted approximation)
- Streaming pipeline (synthesizeStreaming) works with new engine
- WAV output via AVAudioFile (not SherpaOnnxWriteWave)
- Chinese text logs warning and uses English voice (graceful degradation)
  </done>
  </task>

<task type="auto">
  <name>Task 3: Clean up CSherpaOnnx module and validate build</name>
  <files>
    plugins/claude-tts-companion/Sources/CSherpaOnnx/
  </files>
  <action>
1. Delete the entire `Sources/CSherpaOnnx/` directory (contains `empty.c`, `include/` with c-api.h and module.modulemap)
2. Verify no remaining imports of CSherpaOnnx anywhere in the codebase: `grep -r "CSherpaOnnx\|SherpaOnnx\|sherpa.onnx\|sherpa_onnx" plugins/claude-tts-companion/Sources/`
3. If any references remain, remove them
4. Run `swift build -c release` to confirm clean release build
5. Check binary size: `ls -lh .build/release/claude-tts-companion` — expect ~15-25MB stripped
6. Log the build time for reference

**Important:** The metallib file (`mlx.metallib`, 102MB) must be present next to the binary at runtime. This is NOT part of the build step — it's a runtime deployment concern. The executor should note this in the summary but NOT try to solve it here. The existing deployment at `~/.local/bin/` will need the metallib copied alongside the binary.
</action>
<verify>
<automated>cd plugins/claude-tts-companion && swift build -c release 2>&1 | tail -5 && ls -lh .build/release/claude-tts-companion</automated>
</verify>
<done>

- CSherpaOnnx directory deleted
- No sherpa-onnx references remain in source
- Release build succeeds
- Binary size logged
  </done>
  </task>

</tasks>

<verification>
1. `swift build -c release` in plugins/claude-tts-companion/ succeeds
2. No references to CSherpaOnnx, SherpaOnnx, or libonnxruntime in source
3. Package.swift has kokoro-ios, mlx-swift, MLXUtilsLibrary dependencies
4. TTSEngine imports KokoroSwift and uses KokoroTTS
5. Config points to MLX safetensors model path
</verification>

<success_criteria>

- Clean release build with kokoro-ios MLX backend
- Public API preserved: synthesize, synthesizeWithTimestamps, synthesizeStreaming, play, stopPlayback
- Native word timestamps from MToken.start_ts/end_ts replace character-weighted approximation
- No sherpa-onnx code or static library references remain
- Chinese text handled gracefully (warning + English voice fallback)
  </success_criteria>

<output>
After completion, create `.planning/quick/260327-d2e-replace-sherpa-onnx-with-kokoro-ios-mlx-/260327-d2e-SUMMARY.md`
</output>
