# ADR-0033: `candle` as the Production ASR Runtime

## Status

🔄 Superseded by [ADR-0038](adr-0038.md) (2026-06-13)

Voice functionality was extracted to the [omni-voice](https://github.com/rust-works/omni-voice)
repository and removed from omni-dev. The decision below is retained as a
historical record and continues to apply within omni-voice. See ADR-0038.

## Context

[ADR-0032](adr-0032.md) defined the `AudioInput` / `Transcriber` seam in
#801 and landed a `MockTranscriber` placeholder. The "real" Whisper batch
backend originally scoped in #801 was deferred when the no-C++ constraint
ruled out `whisper-rs` (which needs `cmake` + a C++ toolchain). #802 is
the deferred half: pick a runtime, build the backend, ship it.

Spike #813 (branch `issue-813-asr-runtime-spike-candle`, captured in
`SPIKE.md`) prototyped one candidate, **`candle`** (HuggingFace's
pure-Rust ML framework), against the Day-1 stop rule from the issue: if
the first candidate meets every acceptance bar, ship it without
prototyping the others.

The acceptance bars and `candle`'s measured results on the committed
fixture [tests/fixtures/voice/short_en.wav](../../tests/fixtures/voice/short_en.wav)
(11.7 s, 16 kHz mono 16-bit PCM):

| Bar                                  | Result on `openai/whisper-tiny.en` (non-quantized safetensors)       |
| ------------------------------------ | -------------------------------------------------------------------- |
| Transcript accuracy                  | Byte-identical to whisper.cpp baseline; all 7 content words present  |
| Inference latency (warm CPU, greedy) | ~1.16 s on 11.7 s audio (RTF ≈ 0.10)                                 |
| Model load (mmap + struct init)      | ~80 ms                                                               |
| Binary-size delta vs trivial bin     | ~5.5 MiB (LTO=thin, codegen-units=1)                                 |
| `cargo tree \| grep -iE 'cmake\|cc-build\|c\+\+'` | Empty — passes literal acceptance grep                  |
| Cold build time                      | 1 m 33 s                                                             |
| Determinism (3 warm runs)            | Byte-identical transcripts                                           |

`candle` passed every measured bar. Per the Day-1 rule, `tract-onnx`,
`rten`, and `burn` were not prototyped; SPIKE.md records the per-candidate
skip rationale and estimated cost-to-evaluate so the decision can be
revisited cheaply if needed.

## Decision

The production ASR backend uses `candle-core`, `candle-nn`, and
`candle-transformers` at version `0.10.x`, loading
`openai/whisper-tiny.en` (revision `refs/pr/15`) from disk as
`config.json` + `tokenizer.json` + `model.safetensors`.

The backend lives at [src/voice/backends/candle.rs](../../src/voice/backends/candle.rs)
and is selected via `--backend whisper-candle` (or
`OMNI_DEV_VOICE_BACKEND=whisper-candle`). The default backend remains
`mock` until the model storage convention has been through at least one
release cycle — the real backend requires an explicit opt-in.

The 80-bin mel-filter coefficients are vendored from candle's own
example as a 64 KB `include_bytes!`'d blob at
[src/voice/backends/candle_melfilters.bytes](../../src/voice/backends/candle_melfilters.bytes).
This is **data**, not code, and avoids both a build-time codegen step
and a runtime file lookup.

Model files are not downloaded on `voice transcribe`. A separate
one-time CLI command, `omni-dev voice install-model`, fetches them via
`hf-hub` into `~/.omni-dev/voice/models/whisper-tiny.en/`. Path
resolution lives in `src/voice/models.rs` with the priority:

1. `--model <path>` on `voice transcribe` / `voice install-model --dest`.
2. `OMNI_DEV_VOICE_WHISPER_MODEL` env var.
3. The default location above.

## Consequences

### Positive

- **Pure-Rust binary distribution** for the original framing (no C++ /
  cmake) — the literal acceptance grep is empty, and the original
  `whisper-rs` blocker is gone.
- **Single trait surface for batch and streaming.** The `Transcriber`
  trait from #801 already gives `Box<dyn Transcriber>` polymorphism;
  `CandleTranscriber` slots in alongside `MockTranscriber` without
  changes to callers, and the planned `MixdownResampleAudioInput`
  adapter for #806 streaming will compose with it the same way.
- **Install-time failure surfacing.** Downloading the 75 MB
  safetensors at `install-model` rather than on first `transcribe`
  means network failures appear when the user opts in to installing,
  not when they're trying to transcribe captured audio.
- **Atomic, idempotent install.** `voice install-model` writes via
  `.<name>.part` sibling + rename, so a partial download never leaves
  half-written model files in the destination, and re-running on a
  fully-installed model is a no-op (`--force` redownloads).

### Negative

- **Transitive C dependency.** `candle-core` 0.10.2 hard-codes
  `tokenizers = { features = ["onig"] }`. Cargo unions features across
  the dep graph, so `onig_sys` (a C library, not C++) gets compiled
  via the `cc` crate even with `default-features = false` on our
  direct `tokenizers` dep. `ring` (transitively via `rustls` via
  `hf-hub`'s `rustls-tls` feature) also pulls `cc`. **Builds require
  a working C toolchain.**

  The literal acceptance grep
  (`cargo tree | grep -iE 'cmake\|cc-build\|c\+\+'`) is still empty,
  and neither `onig` nor `ring` involves C++, libstdc++/libc++, or
  cmake — so the original `whisper-rs` motivation (heavy C++ + cmake
  failure surface) is genuinely addressed. But the framing of
  "no compiled deps at all" is not technically true.

  Three remediation paths are documented in SPIKE.md if the no-C
  invariant ever becomes hard:
  1. `[patch.crates-io]` redirect to a candle fork that drops the
     `onig` tokenizers feature.
  2. Upstream a PR to `huggingface/candle` to make `onig` non-default
     in `candle-core`'s tokenizers spec.
  3. Swap `rustls-tls` for `native-tls` (trades one form of native
     dep for another; doesn't actually help portability).

  None are needed today.

  [ADR-0037](adr-0037.md) later scopes a narrow, opt-in exception to this
  "no first-party native engine" framing: a pure-**C** engine (`cc` + `make`,
  no C++, no cmake) is permitted behind a Rust FFI boundary on
  `cfg(not(target_os = "windows"))` for the native Voxtral backend. `candle`
  remains the default and the sole Windows ASR backend; this ADR's decision is
  unchanged.

- **Cold-build cost.** `candle-{core,nn,transformers}` add roughly
  1 m to a clean `cargo build`. Incremental builds are unaffected.
  Worth it for the runtime quality; would not be worth it for an
  optional or rarely-used feature.

- **Model directory is opaque outside of `~/.omni-dev/voice/models/`.**
  The convention is "self-contained directory per variant"; users who
  want to put weights elsewhere have to pass `--model` (or set the env
  var) every time. The trade-off is operability — predictable, single
  default — over flexibility.

### Neutral

- **`tiny.en` only for v1.** `base.en`, multilingual, and quantized
  variants are deferred. The `install-model` CLI has no `--variant`
  flag yet; adding it later is an additive change. Future variants
  drop into sibling subdirectories of `~/.omni-dev/voice/models/`.
- **`Mutex<Whisper>` is correctness-critical, not ceremony.** The
  decoder's KV cache is per-segment internal state; two concurrent
  `transcribe` calls on the same backend instance would corrupt it.
  The mutex is held only across the duration of one transcribe, so
  there is no fine-grained contention.
- **Confidence is geometric-mean per-token probability.** Computed as
  `avg_logprob.exp().clamp(0.0, 1.0)`, where `avg_logprob` is the mean
  log-probability of greedy-decoded tokens. Matches whisper.cpp's
  interpretation; not a sigmoid, which would compress probabilities
  into a non-physical range.

## Out of Scope (Tracked Elsewhere)

- **Streaming pipeline** — #806.
- **Word-level alignment** — defer to a later issue (would re-enable
  Whisper timestamps mode and parse per-token timestamps).
- **Speaker diarisation** — #805.
- **CI job that pre-stages the model and runs `--ignored` tests** — a
  follow-up issue. The integration test at
  [tests/voice_transcribe_candle_test.rs](../../tests/voice_transcribe_candle_test.rs)
  is `#[ignore]`-by-default and runs locally via
  `cargo test --test voice_transcribe_candle_test -- --ignored`
  after `omni-dev voice install-model`.
- **Flipping the factory default from `mock` to `whisper-candle`** —
  not until at least one release with the install convention in the
  wild, so first-time users without an installed model see a clear
  CLI error and not a confusing missing-file at `transcribe`.

## Coverage expectations

The seam this ADR establishes has a direct, quantifiable consequence
for line coverage: roughly 175 lines of model-bound code in
`CandleTranscriber` and ~45 lines of network-bound code in
`InstallModelCommand` cannot run in CI without staging the Whisper
tiny.en model on the runner. The `#[ignore]`-gated
`whisper_candle_transcribes_short_en_with_content_words` integration
test exercises the inference path locally after
`omni-dev voice install-model`, but coverage tools record it as
uncovered by default.

This mirrors the by-design gaps in [ADR-0031](adr-0031.md) for cpal
hardware-bound code.

The following gaps are **by design** per this ADR and should not be
chased with additional tests:

| Surface                                                                          | Why uncovered                                                                                                       |
| -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `CandleTranscriber::new` (config parse, tokenizer load, weights mmap, model load) | Needs the staged Whisper config + tokenizer + safetensors on disk; the ensure_model_present error path *is* covered. |
| `CandleTranscriber::transcribe` (mel build, encoder/decoder loop, sampling)      | Needs a real Whisper model in memory.                                                                               |
| `build_suppress_tensor`, `token_id`                                              | Called only by `new()` after the model-present check succeeds.                                                      |
| `InstallModelCommand::download_into` (hf-hub fetch + atomic install)             | Hits HuggingFace's network; running it in CI would couple the build to a third-party service.                       |
| `InstallModelCommand::execute` (stderr-locking wrapper)                          | Forwards to `run()`; only reachable via the network path the caller is opting into.                                  |
| `VoiceSubcommands::InstallModel` dispatch                                        | Same — its only branch delegates to `InstallModelCommand::execute`.                                                  |
| `atomic_install` unreachable defensive arms (`to.parent().is_none()`, `to.file_name().is_none()`) | Both arms are dead-on-absolute-paths; `--dest` is always absolute in practice.                       |
| `default_whisper_model_dir()` `dirs::home_dir() == None` branch                  | Would require unsetting `HOME` globally; racy under parallel tests.                                                 |
| `resolve_whisper_model_dir()` home-dir-missing arm                               | Same as above — depends on the unreachable branch in `default_whisper_model_dir`.                                    |

All *other* code paths in
`src/voice/{backends/candle,factory,models}.rs` and
`src/cli/voice/install_model.rs` should have unit tests driven by
in-process fixtures (`TempDir`, staged-but-empty model dirs,
in-memory `VoiceOpts`). When a coverage gap appears in those files
that isn't on the list above, the right response is to add a test,
not to extend this exception list.

## References

- Issue [#802](https://github.com/rust-works/omni-dev/issues/802) — the
  deferred half of the original "voice transcribe (file-batch mode)"
  scope.
- Issue [#813](https://github.com/rust-works/omni-dev/issues/813) —
  ASR runtime spike that produced this decision.
- [ADR-0031](adr-0031.md) — `AudioSource` (hardware-capture seam).
- [ADR-0032](adr-0032.md) — `AudioInput` / `Transcriber` (ASR seam).
- [ADR-0037](adr-0037.md) — scopes a pure-C / non-Windows / FFI-boundary
  exception to this ADR's no-first-party-native-engine framing.
