voxtral-micro 1.0.0

# Voxtral Micro

A minimal Rust library for text-to-speech using Voxtral Q4-quantized GGUF models.

> This is a reduced version of [voxtral-mini-realtime-rs](https://github.com/TrevorS/voxtral-mini-realtime-rs) for tts only created originally by TrevorS .

## Features

- **Lightweight**: TTS-only functionality, no ASR dependencies
- **Q4 Quantization**: Efficient inference with GGUF models (~2.7 GB)
- **GPU Accelerated**: Metal/Vulkan support via Burn framework
- **20 Voice Presets**: Multiple languages (English, French, German, Spanish, Italian, Portuguese, Dutch, Hindi, Arabic)
- **Speed Control**: Adjust playback speed (0.5× to 3.0×, where 1.0 = normal)
- **Volume Control**: Adjust audio gain (0.1× to 2.0×)
- **Adjustable Quality**: Configure Euler ODE steps (3=fast, 8=quality)

## Installation

Add to your `Cargo.toml`:

```toml
[dependencies]
voxtral-micro = "1.0.0"
```

## Quick Start

### Download Required Files

**All three components must be downloaded:**

```bash
make download-models
```

This downloads and automatically cleans up:
1. **GGUF model** - Core TTS weights (`voxtral-tts-q4.gguf` ~2.7 GB)
2. **Voice embeddings** - 20 voice presets (`.safetensors files` ~17 MB)
3. **Tokenizer** - Text encoding (`tekken.json` ~14 MB)

<details>
<summary>Manual download (if make doesn't work)</summary>

```bash
# Download everything from the TTS Q4 GGUF repo
uv run --with huggingface_hub hf download \
  TrevorJS/voxtral-tts-q4-gguf \
  --local-dir models

# Clean up unnecessary files (optional but recommended)
make clean-models  # Removes shard files, saves 2.7 GB
```
</details>

### Run Examples

**Generate all 20 voices**:
```bash
# Generate all 20 voice samples (saves to voice_samples/)
cargo run --example simple_tts --features "wgpu,native-tokenizer"

# Generate all voices AND play them
cargo run --example simple_tts --features "wgpu,native-tokenizer" -- --play

# Using make (no playback)
make example
```

**Speed and gain control demo**:
```bash
# Generate samples at different speeds (0.5x, 0.75x, 1.0x, 1.5x, 2.0x) and volumes
cargo run --example speed_demo --features "wgpu,native-tokenizer"
```

**Play a single voice**:
```bash
cargo run --example play_voice --features "wgpu,native-tokenizer" -- casual_female
```

### Use as Library

```rust
use voxtral_micro::TtsEngine;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize TTS engine
    let mut tts = TtsEngine::new("models/voxtral-tts-q4.gguf").await?;

    // Synthesize speech
    let audio = tts.synthesize("Hello world!", None)?;

    // Save to WAV file
    tts.save_wav("output.wav", &audio)?;

    Ok(())
}
```

## Advanced Usage

### Custom Voice, Speed, and Volume

```rust
let audio = tts.synthesize_with_options(
    "Bonjour le monde!",
    Some("fr_female"),  // Voice (None = default "casual_female")
    1.5,                // Speed: 1.5 = 50% faster (range: 0.5 to 3.0)
    0.8,                // Gain: 0.8 = 20% quieter (range: 0.1 to 2.0)
    Some("fr")          // Language code
)?;
```

**Speed control**:
- `0.5` = Half speed (slower, lower pitch)
- `1.0` = Normal speed
- `1.5` = 1.5× faster (higher pitch)
- `2.0` = Twice as fast (higher pitch)
- `3.0` = Three times as fast (much higher pitch)

⚠️ **Note**: Speed adjustment uses resampling which changes both tempo and pitch. Higher speeds result in higher pitch (chipmunk effect), lower speeds result in lower pitch.

**Gain control**:
- `0.5` = Half volume (quieter)
- `1.0` = Normal volume
- `1.5` = 1.5× louder
- `2.0` = 2× louder (maximum)

### Adjust Quality/Speed Tradeoff

The TTS engine uses Euler ODE solver with **5 steps by default** (balanced quality/speed). You can adjust this for different use cases:

```rust
// Default: 5 steps (balanced quality and speed)

// Fast mode: 3 steps (faster but may have artifacts like echo, volume drops)
tts.set_euler_steps(3);
let audio = tts.synthesize("Quick response", None)?;

// High quality: 8 steps (best quality, slower generation)
tts.set_euler_steps(8);
let audio = tts.synthesize("High quality speech", None)?;
```

**Performance comparison** (approximate on modern GPU):
- **3 steps**: ~3-4s per voice - Fast but may have artifacts (echo, noise, volume fluctuations)
- **5 steps** (default): ~5-6s per voice - Good quality with reasonable speed
- **8 steps**: ~8-10s per voice - Highest quality, minimal artifacts

⚠️ **Note**: Using fewer than 5 steps may introduce audio artifacts like echo, volume drops at phrase endings, or background noise.

### List Available Voices

```rust
let voices = tts.list_voices()?;
for voice in voices {
    println!("{}", voice);
}
```

## Performance Optimizations

Voxtral Micro is optimized for real-time TTS with the following automatic optimizations:

### Automatic Optimizations (Enabled by Default)

1. **Projection Fusion** (native platforms only):
   - Fuses QKV projections (3→1 matmul per layer)
   - Fuses gate+up projections (2→1 matmul per layer)
   - **Savings**: 87 fewer GPU kernel launches per audio frame
   - Applied automatically on model load (26 backbone + 3 FM layers)

2. **Batched Classifier-Free Guidance**:
   - Runs conditional + unconditional FM passes in single batch
   - Halves the number of FM transformer calls per Euler step

3. **Default 5 Euler Steps**:
   - Balanced configuration for good quality and reasonable speed
   - Reduces artifacts (echo, volume drops, noise) compared to 3 steps
   - Use 3 steps for fastest generation or 8 for highest quality (see "Adjust Quality/Speed Tradeoff")

### Expected Performance

| Configuration | Speed | Quality | Use Case |
|---------------|-------|---------|----------|
| 3 steps | ~3-4s/voice | Fair (artifacts) | Maximum speed, interactive demos |
| 5 steps (default) | ~5-6s/voice | Good | Balanced quality/speed |
| 8 steps | ~8-10s/voice | Best | Production quality, offline generation |

*Benchmarks approximate, vary by GPU (Metal/Vulkan/WGPU backend)*

## Available Voices

| Voice | Language |
|-------|----------|
| casual_female, casual_male | English |
| neutral_female, neutral_male | English |
| cheerful_female | English |
| fr_female, fr_male | French |
| de_female, de_male | German |
| es_female, es_male | Spanish |
| it_female, it_male | Italian |
| pt_female, pt_male | Portuguese |
| nl_female, nl_male | Dutch |
| hi_female, hi_male | Hindi |
| ar_male | Arabic |

## Available Commands

All commands are available via `make`:

```bash
# Run TTS example
make example

# Run with optimizations (faster)
make example-release

# Speed and volume demo
make speed-demo

# Play a specific voice
make play VOICE=casual_female

# Download models (auto-cleans unnecessary files)
make download-models

# Clean up shard files if already downloaded
make clean-models

# Check if models are present
make check-models

# Build library
make build
make build-release

# Development
make check
make test
make fmt
make clippy

# See all commands
make help
```

## Expected Directory Structure

After downloading models:

```
models/
├── voxtral-tts-q4.gguf              # Q4 TTS model (~2.7 GB)
├── tekken.json                       # Tokenizer (~14 MB)
└── voice_embedding/                  # 20 voice presets
    ├── casual_female.safetensors
    ├── casual_male.safetensors
    ├── fr_female.safetensors
    └── ...
```

## API Reference

### `TtsEngine`

#### Methods

- `new(gguf_path) -> Result<TtsEngine>` - Initialize from GGUF model
- `synthesize(text, voice) -> Result<AudioBuffer>` - Synthesize speech (default voice: casual_female)
- `synthesize_with_options(text, voice, speed, gain, language) -> Result<AudioBuffer>` - Synthesize with custom speed/volume
- `list_voices() -> Result<Vec<String>>` - List available voice presets
- `set_euler_steps(steps)` - Set quality (3=fast, 8=quality, default=8)
- `set_max_frames(frames)` - Set maximum audio frames (default: 2000)
- `save_wav(path, audio) -> Result<()>` - Save audio to WAV file

### `AudioBuffer`

Audio data at 24kHz sample rate.

#### Fields

- `samples: Vec<f32>` - Normalized audio samples [-1.0, 1.0]
- `sample_rate: u32` - Sample rate (24000 Hz)

#### Methods

- `len() -> usize` - Number of samples
- `duration_secs() -> f32` - Duration in seconds
- `duration_ms() -> f32` - Duration in milliseconds
- `with_speed(speed) -> AudioBuffer` - Change playback speed (0.5 to 3.0)
- `with_gain(gain) -> AudioBuffer` - Change volume (0.1 to 2.0)
- `save(path) -> Result<()>` - Save to WAV file

## Troubleshooting

### Error: Voice 'casual_female' not found

This means the voice embeddings haven't been downloaded. The GGUF model alone is not sufficient - you need the separate voice embedding files.

**Solution:**
```bash
make download-models  # Downloads all three required components
```

### Error: GGUF model not found

Download the model:
```bash
uv run --with huggingface_hub hf download \
  TrevorJS/voxtral-mini-realtime-gguf voxtral-q4.gguf \
  --local-dir models
```

### Error: Tokenizer not found

Download the tokenizer:
```bash
uv run --with huggingface_hub hf download \
  mistralai/Voxtral-Mini-4B-Realtime-2602 tekken.json \
  --local-dir models/voxtral-tts
```

### Download failed with "Entry Not Found"

Make sure you're using the `--include` flag for the voice embeddings directory:
```bash
# Correct (downloads the voice_embedding directory):
uv run --with huggingface_hub hf download \
  mistralai/Voxtral-4B-TTS-2603 \
  --include "voice_embedding/*" \
  --local-dir models/voxtral-tts

# Wrong (will fail with 404):
# hf download mistralai/Voxtral-4B-TTS-2603 voice_embedding ...
```

### Check what's missing

```bash
make check-models  # Shows which components are present/missing
```

## FAQ

**Q: Why are voices separate from the GGUF model?**

A: Voice embeddings are small (2-4 MB each) compared to the model weights (2.5 GB). Keeping them separate allows you to add custom voices without rebuilding the entire model.

**Q: Can I use this with only the GGUF file?**

A: No, you need all three components: GGUF model, voice embeddings, and tokenizer. Use `make download-models` to get everything.

**Q: How much disk space do I need?**

A: Approximately 2.73 GB total:
- GGUF model: ~2.7 GB
- Voice embeddings: ~17 MB (20 voices)
- Tokenizer: ~14 MB

The download process automatically removes duplicate shard files (~2.7 GB) that are only needed for WASM/browser use.

**Q: What are the shard files?**

A: Shard files (shard-aa, shard-ab, etc.) are the GGUF model split into 512MB chunks for WASM/browser use (browsers can't load files >2GB). For native library use, they're duplicates and automatically removed by `make download-models`. If you downloaded manually, run `make clean-models` to remove them and save 2.7 GB.

**Q: Does speed control change the pitch?**

A: Yes, speed adjustment using resampling changes both tempo and pitch (faster = higher pitch). The library uses a 0.65× scale factor to reduce this effect. For example, user speed `2.0` results in actual `1.3×` playback, which sounds more natural than direct `2.0×` resampling.

## License

Apache-2.0

## Credits

Original implementation by Trevor Strieber. This is a minimal TTS-only fork focused on library usage.