speech-swift — AI 语音合成工具中文文档

Requirements

• Swift 6+, Xcode 16+ (with Metal Toolchain)
• macOS 15+ (Sequoia) or iOS 18+, Apple Silicon (M1/M2/M3/M4)

The macOS 15 / iOS 18 minimum comes from MLState — Apple's persistent ANE state API used by the CoreML pipelines (Qwen3-ASR, Qwen3-Chat, Qwen3-TTS) to keep KV caches resident on the Neural Engine across token steps.

Installation

Build from source

git clone https://github.com/soniqo/speech-swift
cd speech-swift
make build

make build compiles the Swift package and the MLX Metal shader library. The Metal library is required for GPU inference — without it you'll see Failed to load the default metallib at runtime. make debug for debug builds, make test for the test suite.

Full build and install guide →

Quick start

Add the package to your Package.swift:

.package(url: "https://github.com/soniqo/speech-swift", branch: "main")

Import only the modules you need — every model is its own SPM library, so you don't pay for what you don't use:

.product(name: "ParakeetStreamingASR", package: "speech-swift"),
.product(name: "SpeechUI",             package: "speech-swift"),  // optional SwiftUI views

Transcribe an audio buffer in 3 lines:

import ParakeetStreamingASR

let model = try await ParakeetStreamingASRModel.fromPretrained()
let text = try model.transcribeAudio(audioSamples, sampleRate: 16000)

Live streaming with partials:

for await partial in model.transcribeStream(audio: samples, sampleRate: 16000) {
    print(partial.isFinal ? "FINAL: \(partial.text)" : "... \(partial.text)")
}

SwiftUI dictation view in ~10 lines:

import SwiftUI
import ParakeetStreamingASR
import SpeechUI

@MainActor
struct DictateView: View {
    @State private var store = TranscriptionStore()

    var body: some View {
        TranscriptionView(finals: store.finalLines, currentPartial: store.currentPartial)
            .task {
                let model = try? await ParakeetStreamingASRModel.fromPretrained()
                guard let model else { return }
                for await p in model.transcribeStream(audio: samples, sampleRate: 16000) {
                    store.apply(text: p.text, isFinal: p.isFinal)
                }
            }
    }
}

SpeechUI ships only TranscriptionView (finals + partials) and TranscriptionStore (streaming ASR adapter). Use AVFoundation for audio visualization and playback.

Available SPM products: Qwen3ASR, WhisperASR, MossTranscribe, Qwen3TTS, Qwen3TTSCoreML, ParakeetASR, ParakeetStreamingASR, NemotronStreamingASR, OmnilingualASR, CohereTranscribeASR, VoxtralASR, KokoroTTS, SupertonicTTS, VibeVoiceTTS, CosyVoiceTTS, VoxCPM2TTS, IndexTTS2TTS, F5TTS, HiggsTTS, ChatterboxTTS, OmniVoiceTTS, IndicMioTTS, FishAudioTTS, MagpieTTS, MagpieTTSCoreML, MAGNeTMusicGen, StableAudio3MusicGen, FlashSR, PersonaPlex, VoiceChat, CSM, Audio2Face3D, HibikiTranslate, MADLADTranslation, SpeechVAD, SpeechLanguageID, SpeechWakeWord, SpeechEnhancement, SpeechRestoration, SourceSeparation, Qwen3Chat, FunctionGemma, SpeechCore, SpeechUI, AudioCommon.

Code examples

The snippets below show the minimal path for each domain. Every section links to a full guide on soniqo.audio with configuration options, multiple backends, streaming patterns, and CLI recipes.

Speech-to-Text — [full guide →](https://soniqo.audio/guides/transcribe)

import Qwen3ASR

let model = try await Qwen3ASRModel.fromPretrained()
let text = model.transcribe(audio: audioSamples, sampleRate: 16000)

Alternative backends: WhisperASR (Whisper Large-v3 Turbo, native CoreML), Parakeet TDT (CoreML, 32× realtime), Omnilingual ASR (1,672 languages, CoreML or MLX), Streaming dictation (live partials).

Forced Alignment — [full guide →](https://soniqo.audio/guides/align)

import Qwen3ASR

let aligner = try await Qwen3ForcedAligner.fromPretrained()
let aligned = aligner.align(
    audio: audioSamples,
    text: "Can you guarantee that the replacement part will be shipped tomorrow?",
    sampleRate: 24000
)
for word in aligned {
    print("[\(word.startTime)s - \(word.endTime)s] \(word.text)")
}

Text-to-Speech — [full guide →](https://soniqo.audio/guides/speak)

import Qwen3TTS
import AudioCommon

let model = try await Qwen3TTSModel.fromPretrained()
let audio = model.synthesize(text: "Hello world", language: "english")
try WAVWriter.write(samples: audio, sampleRate: 24000, to: outputURL)

Alternative TTS engines: CosyVoice3 (streaming + voice cloning + emotion tags), Kokoro-82M (iOS-ready, 54 voices), VibeVoice (long-form podcast / multi-speaker, EN/ZH), Fish Audio S2 Pro (experimental zero-shot cloning + bracket style markers), Voice cloning.

Speech-to-Speech — [full guide →](https://soniqo.audio/guides/respond)

import PersonaPlex

let model = try await PersonaPlexModel.fromPretrained()
let responseAudio = model.respond(userAudio: userSamples)
// 24 kHz mono Float32 output ready for playback

LLM Chat — [full guide →](https://soniqo.audio/guides/chat)

import Qwen3Chat
import FunctionGemma

let chat = try await Qwen35MLXChat.fromPretrained()
chat.chat(messages: [(.user, "Explain MLX in one sentence")]) { token, isFinal in
    print(token, terminator: "")
}

Translation — [full guide →](https://soniqo.audio/guides/translate)

import MADLADTranslation

let translator = try await MADLADTranslator.fromPretrained()
let es = try translator.translate("Hello, how are you?", to: "es")
// → "Hola, ¿cómo estás?"

Speech Translation — [full guide →](https://soniqo.audio/guides/audio-translate)

import HibikiTranslate
import AudioCommon

let model = try await HibikiTranslateModel.fromPretrained()
let pcm = try AudioFileLoader.load(url: input, targetSampleRate: 24000)
let (englishAudio, textTokens) = model.translate(
    sourceAudio: pcm, sourceLanguage: .fr
)
// Hibiki Zero-3B — FR/ES/PT/DE → EN, on-device, streaming Mimi codec

Voice Activity Detection — [full guide →](https://soniqo.audio/guides/vad)

import SpeechVAD

let vad = try await SileroVADModel.fromPretrained()
let segments = vad.detectSpeech(audio: samples, sampleRate: 16000)
for s in segments { print("\(s.startTime)s → \(s.endTime)s") }

Speaker Diarization — [full guide →](https://soniqo.audio/guides/diarize)

import SpeechVAD

let diarizer = try await DiarizationPipeline.fromPretrained()
let segments = diarizer.diarize(audio: samples, sampleRate: 16000)
for s in segments { print("Speaker \(s.speakerId): \(s.startTime)s - \(s.endTime)s") }

Speech Enhancement — [full guide →](https://soniqo.audio/guides/denoise)

import SpeechEnhancement

let denoiser = try await SpeechEnhancer.fromPretrained()             // CoreML (Neural Engine)
// let denoiser = try await SpeechEnhancer.fromPretrained(engine: .mlx)  // MLX (GPU, fp32)
let clean = try denoiser.enhance(audio: noisySamples, sampleRate: 48000)

Acoustic Echo Cancellation — [full guide →](https://soniqo.audio/guides/echo-cancellation)

import SpeechEnhancement

let aec = try await LocalVQEEchoCanceller.fromPretrained()
let cleanMicrophone = try aec.processFrame(
    microphone: microphoneFrame,
    reference: playbackReferenceFrame
)

Speech Restoration — [full guide →](https://soniqo.audio/guides/restore)

Joint denoise and dereverb with Sidon (w2v-BERT 2.0 predictor + DAC vocoder, Core ML). Unlike a generic noise suppressor, Sidon is trained to preserve speaker identity, so it is well suited to cleaning a noisy or reverberant voice-cloning reference before TTS. Input is 16 kHz; output is 48 kHz mono.

import SpeechRestoration

let restorer = try await SpeechRestorer.fromPretrained()          // .fp16 (default) or .int8
let clean = try restorer.restore(audio: noisySamples, sampleRate: 16000)  // → 48 kHz

From the CLI:

```bash speech restore noisy.wav -o clean.wav # denoise + dereverb, 48 kHz output speech restore noisy.wav --variant int8 # smaller, lower peak RAM

Voice Pipeline (ASR → LLM → TTS) — [full guide →](https://soniqo.audio/voice-agents)

import SpeechCore

let pipeline = VoicePipeline(
    stt: parakeetASR,
    tts: qwen3TTS,
    vad: sileroVAD,
    config: .init(mode: .voicePipeline),
    onEvent: { event in print(event) }
)
pipeline.start()
pipeline.pushAudio(micSamples)

VoicePipeline is the real-time voice-agent state machine (powered by speech-core) with VAD-driven turn detection, interruption handling, and eager STT. It connects any SpeechRecognitionModel + SpeechGenerationModel + StreamingVADProvider.

Demo apps

• DictateDemo (docs) — macOS menu-bar streaming dictation with live partials, VAD-driven end-of-utterance detection, and one-click copy. Runs as a background agent (Parakeet-EOU-120M + Silero VAD).
• iOSEchoDemo — iOS echo demo (Parakeet ASR + Kokoro TTS). Device and simulator.
• PersonaPlexDemo — Conversational voice assistant with mic input, VAD, and multi-turn context. macOS. RTF ~0.94 on M2 Max (faster than real-time).
• Soniqo VoiceChat CLI (guide) — Native full-duplex Nemotron 11B terminal assistant with live RNN-T captions, model-driven turn-taking, adaptive EAR-TTS detail, and optional MCP tools. The included Apple Reminders configuration exposes only create, list, and update operations.
• SpeechDemo — Dictation and TTS synthesis in a tabbed interface. macOS.

Run the VoiceChat reminders demo after a release build:

./.build/release/speech voice-chat \
  --model /path/to/voicechat-mlx-int5 \
  --mcp-config Examples/VoiceChatMCP/apple-reminders.json

Add --debug-timeline for phrase, generated-pronunciation-end, and model-decoded tool lifecycle timestamps. It can reveal tool arguments, so keep it out of shared logs. See each app's README or the linked VoiceChat guide for build and runtime details.

Cache configuration

Model weights download from HuggingFace on first use and cache to ~/Library/Caches/qwen3-speech/. Override with QWEN3_CACHE_DIR (CLI) or cacheDir: (Swift API). All fromPretrained() entry points also accept offlineMode: true to skip network when weights are already cached.

Users in mainland China (or anywhere huggingface.co is slow/blocked) can fetch from a mirror by setting HF_ENDPOINT, e.g. export HF_ENDPOINT=https://hf-mirror.com.

See docs/inference/cache-and-offline.md for full details including sandboxed iOS container paths.

Clean a voice-cloning reference before TTS (opt-in; preserves speaker identity):

speech speak "Hello" --engine voxcpm2 --voice-sample ref.wav --clean-reference ```

HTTP API server

speech-server --port 8080

Exposes every model via HTTP REST + WebSocket endpoints, including OpenAI-compatible APIs: a Realtime WebSocket at /v1/realtime and a transcription REST endpoint at /v1/audio/transcriptions. Realtime sessions use explicit input_audio_buffer.commit by default; send session.update with turn_detection.type set to server_vad for Silero-based automatic speech start/end detection, prefix padding, and bounded turn length. See Sources/AudioServer/ and Shared Protocols.

Swift Package Manager

dependencies: [
    .package(url: "https://github.com/soniqo/speech-swift", branch: "main")
]

Import only what you need — every model is its own SPM target:

import Qwen3ASR             // Speech recognition (MLX)
import WhisperASR           // Whisper Large-v3 Turbo (CoreML)
import MossTranscribe       // MOSS transcription with timestamps + speaker labels (CoreML + MLX)
import ParakeetASR          // Speech recognition (CoreML, batch)
import ParakeetStreamingASR // Streaming dictation with partials + EOU
import NemotronStreamingASR // Multilingual streaming ASR with native punctuation (0.6B, 40 langs)
import OmnilingualASR       // 1,672 languages (CoreML + MLX)
import CohereTranscribeASR  // Cohere Transcribe 2B (MLX, 14 languages)
import VoxtralASR           // Voxtral Mini 3B (MLX, 8 languages)
import Qwen3TTS             // Text-to-speech
import CosyVoiceTTS         // Text-to-speech with voice cloning
import VoxCPM2TTS           // 48 kHz TTS with voice cloning + voice design (2B)
import IndexTTS2TTS         // Native MLX voice cloning from reference audio
import F5TTS                // Zero-shot voice cloning (DiT flow matching + Vocos)
import HiggsTTS             // Conversational TTS + cloning (Qwen3 backbone, control tags)
import CSM                  // Conversational Speech Model — text→audio + voice cloning (Sesame CSM-1B, MLX)
import KokoroTTS            // Text-to-speech (iOS-ready)
import VibeVoiceTTS         // Long-form / multi-speaker TTS (EN/ZH)
import MagpieTTS            // Multilingual TTS (NVIDIA Magpie 357M, MLX, 9 langs)
import MagpieTTSCoreML      // Magpie CoreML backend (hybrid CoreML + MLX, 8 langs)
import FishAudioTTS         // Experimental Fish Audio S2 Pro runtime with voice cloning
import IndicMioTTS          // Hindi/Indic TTS with emotion markers
import Qwen3Chat            // On-device LLM chat
import FunctionGemma    // On-device tool-call LLM
import MADLADTranslation    // Many-to-many translation across 400+ languages
import HibikiTranslate      // Streaming speech-to-speech translation (FR/ES/PT/DE → EN)
import PersonaPlex          // Full-duplex speech-to-speech
import SpeechVAD            // VAD + speaker diarization + embeddings
import SpeechWakeWord       // Wake-word / keyword spotting
import SpeechEnhancement    // Noise suppression
import SpeechRestoration    // Speech restoration — denoise + dereverb (Sidon, CoreML, 48 kHz)
import SourceSeparation     // Music source separation (Open-Unmix, 4 stems)
import StableAudio3MusicGen // Text-to-audio/music generation (Stable Audio 3)
import SpeechUI             // SwiftUI components for streaming transcripts
import AudioCommon          // Shared protocols and utilities