mozu-audio 0.3.0

<p align="center">
  <img src="assets/mozu-audio-icon.png" alt="Mozu Audio icon" width="144" height="144">
</p>

<h1 align="center">Mozu Audio</h1>

<p align="center">
  A Rust SDK for local speech recognition on macOS, powered by Core ML.
</p>

- Japanese speech recognition with Haas.
- Long-audio transcription with incremental WAV reading and bounded audio buffering.
- Streaming sessions with provisional, confirmed, and final transcript updates.
- Silero VAD, microphone capture, resampling, token timestamps, and cancellation.
- Models stay loaded across recognition calls. Audio is processed locally.

## At a Glance

| Capability | Model | Speed (RTFx) | Accuracy |
| --- | --- | ---: | ---: |
| Japanese read-speech transcription | Haas-ja v1.0 Core ML · 15 s | 212× | 3.98% CER |
| Japanese long-form conversation | Haas-ja v1.0 Core ML · 30.72 s | 246× | 8.54% CER |

Measured with Mozu Audio 0.1.0 on Apple M3 (macOS 26.6.2), using INT8 encoder weights and the FP32 split decoder. Read speech uses 479 clips from the Japanese FLEURS test split (91 minutes), with speed based on the mean of two passes. Long-form transcription uses quiet-point splitting on the nine J-CRe3 test recordings (23 minutes), with speed based on the sum of per-recording medians over four passes. RTFx is audio duration divided by processing time. Timings exclude model loading, warm-up, and file I/O. CER is character error rate after text normalization.

## Requirements

- macOS on Apple Silicon.
- Rust 1.94 or later.
- Compatible Haas Core ML models and preprocessing/tokenizer assets.
- A Silero VAD Core ML model for speech-boundary detection.

## Add the library

Add the SDK from crates.io:

```sh
cargo add mozu-audio@0.3.0
```

To use a local checkout:

```sh
cargo add mozu-audio --path /path/to/mozu-audio/crates/core
```

## Models

Use `ModelCache` to download the compiled [Haas Core ML models](https://huggingface.co/ooookao/haas-ja-v1.0-coreml) and [Silero VAD](https://huggingface.co/FluidInference/silero-vad-coreml) into an application-owned directory. A verified cache works offline. Call these blocking methods on a worker thread in GUI applications.

```rust
use mozu_audio::{Cancellation, ComputeConfig, HaasEngine, models::ModelCache};
use std::path::PathBuf;

fn load_haas(cache_dir: PathBuf) -> mozu_audio::Result<HaasEngine> {
    let cache = ModelCache::new(cache_dir);
    let files = cache.prepare_haas(&Cancellation::new(), |progress| {
        eprintln!("{}: {} bytes", progress.file, progress.received_bytes);
    })?;
    files.load(ComputeConfig::standard())
}
```

For VAD, pass the path from `cache.prepare_silero(&cancellation, |_| {})?` to `SileroVad::load`. To use the local-path WAV example below, place `assets/`, `encoder.mlmodelc/`, and `predictor.mlmodelc/` under `models/haas/`.

| Input | Required files |
| --- | --- |
| Haas assets directory | `metadata.json`, `tokenizer.json`, `window.f32`, `filterbank.f32`, `embeddings.f32` |
| Haas encoder | Compatible compiled Core ML model (`.mlmodelc`) |
| Haas predictor/joint | Compatible compiled Core ML model (`.mlmodelc`) |
| Silero VAD | Compatible 256 ms Core ML model with recurrent state (`.mlmodelc`) |

Use `HaasEngine::load` for a fused predictor/joint. For separate models, use `HaasEngine::load_split` with a prediction model and either `joint.mlmodelc` (full logits) or `joint-decision.mlmodelc` (scalar decisions).

Model support currently covers [Haas-ja-v1.0](https://huggingface.co/third-intelligence/Haas-ja-v1.0) and Silero VAD 6.2.1. Original PyTorch checkpoints cannot be passed directly to the runtime.
The exporters are [`tools/convert_haas_coreml.py`](tools/convert_haas_coreml.py) and [`tools/convert_silero_coreml.py`](tools/convert_silero_coreml.py); run them with `--help` for conversion requirements and arguments. Python is needed only when exporting models.

## Transcribe a WAV file

```rust
use mozu_audio::{Cancellation, HaasEngine};
use std::path::Path;

fn main() -> mozu_audio::Result<()> {
    let engine = HaasEngine::load(
        Path::new("models/haas/assets"),
        Path::new("models/haas/encoder.mlmodelc"),
        Path::new("models/haas/predictor.mlmodelc"),
    )?;
    let result = engine.transcribe_wav(
        Path::new("speech.wav"),
        &Cancellation::new(),
    )?;
    println!("{}", result.text);
    Ok(())
}
```

For in-memory audio, use `HaasEngine::transcribe_samples` with 16 kHz mono `f32` samples.
A single call accepts at least 0.64 seconds and up to the loaded model's limit. Read the sample bounds with `engine.input_limits()`. Use the batch or streaming API for longer input.
Keep the engine on the worker thread that created it and reuse it across calls.

## Long-audio transcription

Use `batch::transcribe_quiet_file` to transcribe a recording in ranges split at quiet points:

```rust
use mozu_audio::{Cancellation, HaasEngine, batch};
use std::path::Path;

fn transcribe_file(engine: &HaasEngine, path: &Path) -> mozu_audio::Result<String> {
    let result = batch::transcribe_quiet_file(
        engine,
        path,
        &Cancellation::new(),
        |progress| {
            println!("{} / {} samples", progress.processed_samples, progress.total_samples);
            Ok(())
        },
    )?;
    Ok(result.transcript.text)
}
```

Each range fits the loaded model. Batch results include the transcript, token timestamps, elapsed processing time, and peak audio buffer size.
Use `transcribe_quiet_wav` for canonical 16 kHz mono float32 or PCM16 WAV input, or `transcribe_quiet_source` for a reusable audio source. The `_result` variants return shared ASR metadata and token pieces.
For VAD-based overlapping windows, use `batch::transcribe_file` with a loaded `SileroVad` and `AsrConfig`.

Use `transcribe_quiet_file_with_updates` or `transcribe_file_with_updates` to receive partial text, tokens, and processing progress. The final update contains the assembled transcript.

## Streaming recognition

Create a `streaming::StreamingSession` with a loaded engine, VAD model, and `AsrConfig::interactive_for(engine.input_limits())?`.
Feed 16 kHz mono `f32` audio through `push_audio`, with at most 16,000 samples per call, and consume the returned updates before sending the next block.

| Update field | Meaning |
| --- | --- |
| `kind` | `partial`, `confirmed`, or `final` |
| `session_generation` | Revision lineage, advanced by `reset` |
| `revision` | Retained prefix, replacement tokens, and stable token count |
| `confirmed_text` | Committed transcript prefix |
| `provisional_text` | Uncommitted suffix that may change |
| `confirmed_samples` | Position through which input has been committed |
| `received_samples` | Total input received |

Apply token revisions by retaining `retained_token_count` tokens and appending `appended_tokens`. Tokens before `stable_token_count` are immutable. Process empty revisions as well; a final update makes every remaining token stable.

Call `finish` to flush the remaining input and retrieve the final transcript, or `cancel` to discard the session.
Call `reset` before reusing a completed, cancelled, or failed session. Loaded models are retained.
Microphone capture and native-rate-to-16 kHz resampling are available in `recording`.

Use `AsrService::load_windowed_with_options` or `AsrService::load_with_options` with a `source_capacity` greater than one to share loaded models across independent sources. Start a session for each `SourceId` and use `snapshot` to read its current transcript.

Select allowed Core ML compute units for the encoder and decoder with `ComputeConfig`. Pass it to the engine's `load_with_compute` or `load_split_with_compute`, or set it in `ServiceOptions`. Use `SileroVad::load_with_compute` to select VAD compute units.

Streaming uses overlapping windows over the offline Haas model. It does not provide model-native cached streaming.

## Custom vocabulary

Create `VocabularyTerm` entries with the desired spelling and optional readings. Load the split decoder with a joint model that returns logits, then call `HaasEngine::compile_vocabulary` with the vocabulary and the matching Haas `tokenizer.model`. Use the compiled vocabulary with `transcribe_samples_with_vocabulary`, the batch vocabulary APIs, or `StreamingSession::windowed_with_vocabulary`. For service sessions, supply a `VocabularyRequest` when starting the source.

## CLI

Build the included command-line tool:

```sh
cargo build --release --workspace --locked
```

Download Haas and Silero into a local cache:

```sh
./target/release/mozu download-models "$HOME/Library/Caches/mozu-audio"
```

Append `15s` to download the 1,500-frame Haas encoder instead of the default 3,072-frame encoder.

Record and transcribe from the default microphone:

```sh
./target/release/mozu record models/haas/assets models/haas/encoder.mlmodelc models/haas/predictor.mlmodelc
```

Press Enter to start recording and Enter again to stop and transcribe. Enter `cancel` to discard a recording or `q` to exit.
Microphone access requires permission for the terminal application in macOS settings. Recognition results are written as JSON to standard output.

## API constraints

- Token timestamps are decoder positions at 80 ms resolution, rather than word-level forced alignment.
- Cancellation is cooperative; an individual Core ML prediction already in progress completes before cancellation takes effect.