---
phase: 03-tts-engine
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - plugins/claude-tts-companion/Sources/CSherpaOnnx/include/sherpa-onnx/c-api/c-api.h
  - plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift
  - plugins/claude-tts-companion/Sources/claude-tts-companion/Config.swift
autonomous: true
requirements: [TTS-01, TTS-02, TTS-03, TTS-04, TTS-05, TTS-08]
must_haves:
  truths:
    - "sherpa-onnx static libs expose durations field in SherpaOnnxGeneratedAudio"
    - "TTSEngine synthesizes text to a 24kHz mono 16-bit WAV file"
    - "afplay plays the generated WAV without blocking the main thread"
    - "Model loads lazily on first synthesis call, not at init"
    - "Synthesis runs on a dedicated serial DispatchQueue"
  artifacts:
    - path: "plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift"
      provides: "TTS synthesis engine wrapping sherpa-onnx C API"
      exports: ["TTSEngine"]
    - path: "plugins/claude-tts-companion/Sources/CSherpaOnnx/include/sherpa-onnx/c-api/c-api.h"
      provides: "Updated C API header with durations field"
      contains: "durations"
  key_links:
    - from: "TTSEngine.swift"
      to: "CSherpaOnnx"
      via: "import CSherpaOnnx + SherpaOnnxOfflineTtsGenerate"
      pattern: "SherpaOnnxCreateOfflineTts|SherpaOnnxOfflineTtsGenerate"
    - from: "TTSEngine.swift"
      to: "Config.swift"
      via: "Config.kokoroModelPath for model directory"
      pattern: "Config\\.kokoroModelPath"
---

<objective>
Patch sherpa-onnx to expose duration tensors, rebuild static libs, and create TTSEngine.swift that wraps the C API for Kokoro TTS synthesis with lazy loading, background queue execution, and afplay playback.

Purpose: Establishes the core TTS pipeline -- text goes in, audio comes out on a background thread without blocking UI. The duration tensor exposure enables Plan 02's word-level timestamps.

Output: Patched sherpa-onnx libs with durations support, TTSEngine.swift with synthesize/play API, updated vendored C header.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/02-subtitle-overlay/02-01-SUMMARY.md
@.planning/phases/02-subtitle-overlay/02-02-SUMMARY.md

<interfaces>
<!-- Key types and contracts from existing codebase -->

From Config.swift:

```swift
enum Config {
    static let sherpaOnnxPath: String  // ~/fork-tools/sherpa-onnx/build-swift-macos/install
    static let kokoroModelPath: String // ~/tmp/subtitle-spikes-7aqa/03-textream/models-int8/kokoro-int8-en-v0_19
    static let appName: String
}
```

From CSherpaOnnx c-api.h (current -- will be patched):

```c
typedef struct SherpaOnnxGeneratedAudio {
    const float *samples;  // in the range [-1, 1]
    int32_t n;             // number of samples
    int32_t sample_rate;
    // durations field will be added by Task 1
} SherpaOnnxGeneratedAudio;

const SherpaOnnxOfflineTts *SherpaOnnxCreateOfflineTts(const SherpaOnnxOfflineTtsConfig *config);
void SherpaOnnxDestroyOfflineTts(const SherpaOnnxOfflineTts *tts);
const SherpaOnnxGeneratedAudio *SherpaOnnxOfflineTtsGenerate(
    const SherpaOnnxOfflineTts *tts, const char *text, int32_t sid, float speed);
void SherpaOnnxDestroyOfflineTtsGeneratedAudio(const SherpaOnnxGeneratedAudio *p);
int32_t SherpaOnnxWriteWave(const float *samples, int32_t n, int32_t sample_rate, const char *filename);
```

From SubtitlePanel.swift (downstream consumer in Plan 02):

```swift
@MainActor final class SubtitlePanel: NSPanel {
    func showUtterance(_ text: String, wordTimings: [TimeInterval])
}
```

Sherpa-onnx Kokoro config struct (from c-api.h):

```c
typedef struct SherpaOnnxOfflineTtsKokoroModelConfig {
    const char *model;      // path to model.onnx
    const char *voices;     // path to voices.bin
    const char *tokens;     // path to tokens.txt
    const char *data_dir;   // path to espeak-ng-data/
    float length_scale;
    const char *dict_dir;
    const char *lexicon;
    const char *lang;
} SherpaOnnxOfflineTtsKokoroModelConfig;
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Patch sherpa-onnx to expose duration tensor and rebuild static libs</name>
  <files>
    plugins/claude-tts-companion/Sources/CSherpaOnnx/include/sherpa-onnx/c-api/c-api.h
    plugins/claude-tts-companion/Sources/claude-tts-companion/Config.swift
  </files>
  <read_first>
    - plugins/claude-tts-companion/Sources/CSherpaOnnx/include/sherpa-onnx/c-api/c-api.h
    - plugins/claude-tts-companion/Sources/claude-tts-companion/Config.swift
    - ~/fork-tools/sherpa-onnx/sherpa-onnx/csrc/offline-tts-kokoro-model.cc
    - ~/fork-tools/sherpa-onnx/sherpa-onnx/csrc/offline-tts-kokoro-model.h
    - ~/fork-tools/sherpa-onnx/sherpa-onnx/csrc/offline-tts-kokoro-impl.h
    - ~/fork-tools/sherpa-onnx/sherpa-onnx/csrc/offline-tts.h
    - ~/fork-tools/sherpa-onnx/build-swift-macos/install/include/sherpa-onnx/c-api/c-api.h
    - ~/tmp/subtitle-spikes-7aqa/16-onnx-timestamps-swift/SPIKE-16-ONNX-TIMESTAMPS-REPORT.md
  </read_first>
  <action>
    **Part A: Prepare timestamped model for sherpa-onnx compatibility**

    The standard int8 model at Config.kokoroModelPath has only 1 output (audio). The timestamped model at `~/tmp/subtitle-spikes-7aqa/16-onnx-timestamps-swift/timestamped-model/onnx/model_quantized.onnx` has 2 outputs (waveform + durations) but lacks the ONNX metadata sherpa-onnx needs (model_type, sample_rate, style_dim, etc.).

    Use the `onnxruntime` Python library to copy metadata from the standard model to the timestamped model:

    ```bash
    uv run --python 3.13 --with onnx python3 -c "
    import onnx
    # Load standard model to get metadata
    std = onnx.load('$HOME/tmp/subtitle-spikes-7aqa/03-textream/models-int8/kokoro-int8-en-v0_19/model.int8.onnx')
    # Load timestamped model
    ts = onnx.load('$HOME/tmp/subtitle-spikes-7aqa/16-onnx-timestamps-swift/timestamped-model/onnx/model_quantized.onnx')
    # Copy metadata entries from standard to timestamped
    for entry in std.metadata_props:
        ts.metadata_props.append(entry)
    onnx.save(ts, '$HOME/tmp/subtitle-spikes-7aqa/03-textream/models-int8/kokoro-int8-en-v0_19/model-timestamped.int8.onnx')
    "
    ```

    Then set up the model directory so sherpa-onnx can find all needed files. The existing `tokens.txt`, `voices.bin`, and `espeak-ng-data/` from the standard model directory are compatible -- only the ONNX model file changes.

    Update Config.swift: Add a new `kokoroTimestampedModelFile` constant pointing to `model-timestamped.int8.onnx` in the model directory. Keep the existing `kokoroModelPath` pointing to the directory. Add:

    ```swift
    /// Filename of the timestamped Kokoro model (has duration tensor output)
    static let kokoroModelFile = "model-timestamped.int8.onnx"
    ```

    **Part B: Patch sherpa-onnx C++ source (4 files, ~50 lines)**

    All edits are in `~/fork-tools/sherpa-onnx/`:

    1. **`sherpa-onnx/csrc/offline-tts.h`** -- Add durations to GeneratedAudio struct:
       ```cpp
       struct GeneratedAudio {
           std::vector<float> samples;
           int32_t sample_rate;
           std::vector<float> durations;     // per-token durations from model
           int32_t num_duration_tokens = 0;  // number of duration values
           GeneratedAudio ScaleSilence(float scale) const;
       };
       ```

    2. **`sherpa-onnx/csrc/offline-tts-kokoro-model.h`** -- Change Run return type from `Ort::Value` to `std::pair<Ort::Value, Ort::Value>`:
       Find the `Run` method declaration and change its return type.

    3. **`sherpa-onnx/csrc/offline-tts-kokoro-model.cc`** -- Return both tensors from Run:
       At line 98-102, change:
       ```cpp
       return std::move(out[0]);
       ```
       to:
       ```cpp
       if (out.size() > 1) {
           return {std::move(out[0]), std::move(out[1])};
       }
       return {std::move(out[0]), Ort::Value(nullptr)};
       ```

    4. **`sherpa-onnx/csrc/offline-tts-kokoro-impl.h`** -- At line ~404, capture both outputs:
       ```cpp
       auto [audio, durations_tensor] = model_->Run(std::move(x_tensor), sid, speed);
       ```
       Then after building `ans.samples`, add:
       ```cpp
       if (durations_tensor) {
           auto dur_shape = durations_tensor.GetTensorTypeAndShapeInfo().GetShape();
           int64_t dur_total = 1;
           for (auto d : dur_shape) dur_total *= d;
           const float *dur_p = durations_tensor.GetTensorData<float>();
           ans.durations = std::vector<float>(dur_p, dur_p + dur_total);
           ans.num_duration_tokens = static_cast<int32_t>(dur_total);
       }
       ```

    5. **C API wrapper** -- In `sherpa-onnx/csrc/c-api.cc`, find where `SherpaOnnxGeneratedAudio` is populated from `GeneratedAudio`. Add durations copy:
       ```cpp
       if (!audio.durations.empty()) {
           float *dur = new float[audio.num_duration_tokens];
           std::copy(audio.durations.begin(), audio.durations.end(), dur);
           ans->durations = dur;
           ans->num_durations = audio.num_duration_tokens;
       } else {
           ans->durations = nullptr;
           ans->num_durations = 0;
       }
       ```
       Also update `SherpaOnnxDestroyOfflineTtsGeneratedAudio` to free `durations`.

    **Part C: Update C API header**

    In `~/fork-tools/sherpa-onnx/sherpa-onnx/c-api/c-api.h` (the source), add to `SherpaOnnxGeneratedAudio`:
    ```c
    const float *durations;   // per-token durations (NULL if model doesn't support)
    int32_t num_durations;    // number of duration values
    ```

    **Part D: Rebuild sherpa-onnx static libs**

    ```bash
    cd ~/fork-tools/sherpa-onnx/build-swift-macos
    cmake --build . --config Release -j $(sysctl -n hw.ncpu)
    cmake --install . --prefix install
    ```

    This should take ~3-5 minutes. Verify the new header is installed:
    ```bash
    grep "durations" ~/fork-tools/sherpa-onnx/build-swift-macos/install/include/sherpa-onnx/c-api/c-api.h
    ```

    **Part E: Update vendored header in the project**

    Copy the updated installed header to the project:
    ```bash
    cp ~/fork-tools/sherpa-onnx/build-swift-macos/install/include/sherpa-onnx/c-api/c-api.h \
       plugins/claude-tts-companion/Sources/CSherpaOnnx/include/sherpa-onnx/c-api/c-api.h
    ```

    **Part F: Verify build still works**

    ```bash
    cd plugins/claude-tts-companion && swift build 2>&1 | tail -5
    ```

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/cc-skills/plugins/claude-tts-companion && swift build 2>&1 | tail -3</automated>
  </verify>
  <acceptance_criteria>
    - grep "durations" plugins/claude-tts-companion/Sources/CSherpaOnnx/include/sherpa-onnx/c-api/c-api.h returns "const float *durations" and "int32_t num_durations"
    - grep "durations" ~/fork-tools/sherpa-onnx/sherpa-onnx/csrc/offline-tts.h returns "std::vector<float> durations"
    - grep "kokoroModelFile" plugins/claude-tts-companion/Sources/claude-tts-companion/Config.swift returns the timestamped model filename
    - swift build succeeds with zero errors
    - File exists: ~/tmp/subtitle-spikes-7aqa/03-textream/models-int8/kokoro-int8-en-v0_19/model-timestamped.int8.onnx
  </acceptance_criteria>
  <done>sherpa-onnx static libs rebuilt with durations support, vendored C header updated, timestamped model prepared with metadata, Config.swift has model file constant, project compiles</done>
</task>

<task type="auto">
  <name>Task 2: Create TTSEngine.swift with synthesis, WAV output, and afplay playback</name>
  <files>plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift</files>
  <read_first>
    - plugins/claude-tts-companion/Sources/CSherpaOnnx/include/sherpa-onnx/c-api/c-api.h
    - plugins/claude-tts-companion/Sources/claude-tts-companion/Config.swift
    - plugins/claude-tts-companion/Sources/claude-tts-companion/main.swift
    - plugins/claude-tts-companion/Package.swift
  </read_first>
  <action>
    Create `TTSEngine.swift` with the following implementation:

    ```swift
    import Foundation
    import Logging
    import CSherpaOnnx

    /// Wraps sherpa-onnx Kokoro TTS for speech synthesis with word-level timestamps.
    ///
    /// - Model loads lazily on first `synthesize()` call (TTS-03)
    /// - All synthesis runs on a dedicated serial DispatchQueue (TTS-02)
    /// - Audio written as 24kHz mono 16-bit WAV via SherpaOnnxWriteWave (TTS-08)
    /// - Playback via afplay subprocess (TTS-01)
    final class TTSEngine: @unchecked Sendable {
        // ... implementation
    }
    ```

    **Key design decisions:**

    1. **Lazy model loading (TTS-03)**: `ttsInstance` is `nil` at init. On first `synthesize()` call, create `SherpaOnnxOfflineTts` via `SherpaOnnxCreateOfflineTts`. Store as `UnsafePointer<SherpaOnnxOfflineTts>?`. Use a lock (`NSLock`) to protect the lazy init since the serial queue guarantees serial access but we want thread-safety for the property.

    2. **Serial DispatchQueue (TTS-02)**: Create `private let queue = DispatchQueue(label: "com.terryli.tts-engine", qos: .userInitiated)`. All synthesis work dispatches to this queue. The queue label must match the app's service label pattern.

    3. **@unchecked Sendable**: TTSEngine is Sendable because all mutable state is accessed only from the serial queue. The `@unchecked` annotation is correct for Swift 6.

    4. **Config setup**: Build `SherpaOnnxOfflineTtsConfig` with:
       - `config.model.kokoro.model` = `"\(Config.kokoroModelPath)/\(Config.kokoroModelFile)"` (the timestamped model)
       - `config.model.kokoro.voices` = `"\(Config.kokoroModelPath)/voices.bin"`
       - `config.model.kokoro.tokens` = `"\(Config.kokoroModelPath)/tokens.txt"`
       - `config.model.kokoro.data_dir` = `"\(Config.kokoroModelPath)/espeak-ng-data"`
       - `config.model.kokoro.length_scale` = `1.0`
       - `config.model.num_threads` = `4`
       - `config.model.provider` = `"cpu"`
       - `config.max_num_sentences` = `1`

    5. **C struct initialization**: In Swift, C structs imported from headers have default memberwise inits. Initialize `SherpaOnnxOfflineTtsConfig()` which zero-initializes all fields, then set each field. Use `.withCString` for string fields, storing the C strings in a retained array to prevent deallocation during the call.

    IMPORTANT: C string fields in the config struct are `const char *`. Swift's `.withCString` creates a temporary pointer. You MUST keep the Swift strings alive for the duration of the `SherpaOnnxCreateOfflineTts` call. Pattern:
    ```swift
    let modelPath = "\(Config.kokoroModelPath)/\(Config.kokoroModelFile)"
    let voicesPath = "\(Config.kokoroModelPath)/voices.bin"
    // ... etc
    // Use nested withCString calls or strdup + free pattern
    ```

    6. **Synthesis method**:
       ```swift
       struct SynthesisResult {
           let wavPath: String
           let audioDuration: TimeInterval
           let durations: [Float]?  // raw duration tensor values, nil if model doesn't support
       }

       func synthesize(text: String, speakerId: Int32 = 0, speed: Float = 1.0,
                        completion: @escaping (Result<SynthesisResult, Error>) -> Void)
       ```

       On the serial queue:
       a. Ensure model is loaded (lazy init)
       b. Call `SherpaOnnxOfflineTtsGenerate(tts, text, sid, speed)`
       c. Get audio from result: `result.pointee.samples`, `result.pointee.n`, `result.pointee.sample_rate`
       d. Get durations from result: `result.pointee.durations`, `result.pointee.num_durations` (may be nil/0 if model doesn't support)
       e. Write WAV to temp file: `SherpaOnnxWriteWave(samples, n, sampleRate, tempPath)`
       f. Free result: `SherpaOnnxDestroyOfflineTtsGeneratedAudio(result)`
       g. Compute `audioDuration = Double(n) / Double(sampleRate)`
       h. Return `SynthesisResult` via completion handler on the serial queue

    7. **WAV temp file**: Use `NSTemporaryDirectory() + "tts-\(UUID().uuidString).wav"`. Clean up previous temp file before writing new one.

    8. **afplay playback**:
       ```swift
       func play(wavPath: String, completion: (() -> Void)? = nil)
       ```
       Spawn `Process()` with `/usr/bin/afplay` and the WAV path. Run on the serial queue. Set `terminationHandler` to call completion. Store the Process reference so it can be cancelled.

    9. **Cleanup**: `deinit` calls `SherpaOnnxDestroyOfflineTts` if model was loaded. Also terminate any running afplay process.

    10. **Logging**: Use `private let logger = Logger(label: "tts-engine")`. Log model load time, synthesis duration, audio duration, RTF (real-time factor = synthesis_time / audio_duration).

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/cc-skills/plugins/claude-tts-companion && swift build 2>&1 | tail -3</automated>
  </verify>
  <acceptance_criteria>
    - grep "final class TTSEngine" plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift returns a match
    - grep "SherpaOnnxCreateOfflineTts" plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift returns a match
    - grep "SherpaOnnxOfflineTtsGenerate" plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift returns a match
    - grep "SherpaOnnxWriteWave" plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift returns a match
    - grep "afplay" plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift returns a match
    - grep "DispatchQueue.*tts-engine" plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift returns a match
    - grep "struct SynthesisResult" plugins/claude-tts-companion/Sources/claude-tts-companion/TTSEngine.swift returns a match
    - swift build succeeds with zero errors
  </acceptance_criteria>
  <done>TTSEngine.swift exists with lazy model loading, serial queue synthesis, WAV output via SherpaOnnxWriteWave, afplay playback, and SynthesisResult containing raw durations. Project compiles.</done>
</task>

</tasks>

<verification>
1. `swift build` succeeds with zero errors
2. grep confirms durations field in vendored c-api.h
3. grep confirms TTSEngine.swift has all required C API calls
4. grep confirms lazy loading pattern (ttsInstance initialized on first use)
5. grep confirms serial DispatchQueue for background synthesis
</verification>

<success_criteria>

- sherpa-onnx static libs rebuilt with duration tensor support
- Vendored c-api.h updated with durations/num_durations fields
- Timestamped Kokoro model prepared with sherpa-onnx metadata
- TTSEngine.swift wraps full synthesis pipeline (init -> generate -> WAV -> play)
- All synthesis runs on dedicated serial queue, not main thread
- Model loads lazily on first call
- Project compiles with zero errors
  </success_criteria>

<output>
After completion, create `.planning/phases/03-tts-engine/03-01-SUMMARY.md`
</output>
