rightkit-qts

Native Qwen3-TTS inference for Right Suite apps: GGUF and GGML talker and code-predictor models, a tokenizer GGUF, and a shared 12 Hz ONNX vocoder. It is vendored from QTS v0.2.0 (see PINNED.md). The Rust library name is qts. It owns synthesis (base, custom voice, and voice design modes), streaming, model-path resolution, and the GGUF export contract. It does not own voice selection UX, model downloads, or worker lifetime. rightkit-tts wraps this crate in a worker with idle unload.

Use it through rightkit-tts in Right Suite apps. Use this crate directly only when an app needs the engine without the worker.

Do not use it to download models. Model files must already be on disk; the crate does not fetch them. It is not a speech recognizer.

Features

The default feature set is empty. This crate has no direct ort dependency; the ONNX Runtime pin (=2.0.0-rc.13, load-dynamic, API 24) is owned by rightkit-ort (crates/rightkit-ort/Cargo.toml:59).

Platform support

• metal applies to Apple targets. rightkit-qts-ggml-sys ignores it with a build warning on other targets.
• vulkan is for hosts with a Vulkan SDK.
• The vocoder runs through ONNX Runtime. The execution provider is chosen by feature and by the rightkit-ort binding that the app configures.
• cfg(target_vendor = "apple") branches exist in src/pipeline/backend.rs and src/pipeline/vocoder.rs.

Published version: see INDEX.md at the repository root. Release notes: see CHANGELOG.md.

Compatibility

Existing SynthesizeRequest, ModelPaths, Base x-vector/ICL synthesis, streaming, profiling & 0.6B filename resolution retain their public signatures. Existing synthesize() also retains its zero-embedding fallback. Mode APIs are additive, so existing struct literals need no new fields.

Set RIGHTKIT_QTS_DUMP_CODES to a file path (for example <dir>/qts-codes.json) to capture first 32 generated codec frames as JSON [[codebook_0, ..., codebook_15], ...]. Capture excludes reference frames, persists after each frame, & resets to [] on each rollout. Recompute, cached, & streaming rollouts support it; path errors return IO errors. Unset variable leaves synthesis unchanged.

Example

The Mode API section below shows the call sequence (illustrative). The unit tests named in Synthetic tests are the compiled checks.

Mode API

Illustrative; not compiled as a doctest. The compiled checks are the unit tests named in Synthetic tests.

use qts::{ModeSynthesizeRequest, PresetSpeaker, QtsMode, Qwen3TtsEngine,
          SynthesizeRequest};

let engine = Qwen3TtsEngine::from_model_dir_for_mode(
    "/models/customvoice", QtsMode::CustomVoice,
)?;
let request = ModeSynthesizeRequest {
    synthesis: SynthesizeRequest {
        text: "Welcome!".into(),
        ..Default::default()
    },
    mode: QtsMode::CustomVoice,
    speaker: Some(PresetSpeaker::Vivian),
    instruct: Some("Speak warmly & slowly.".into()),
    ..Default::default()
};
let audio = engine.synthesize_mode(&request)?;
# Ok::<(), qts::Qwen3TtsError>(())

pub enum QtsMode { Base, CustomVoice, VoiceDesign }
pub enum PresetSpeaker {
    Vivian, Serena, Uncle_Fu, Dylan, Eric, Ryan, Aiden, Ono_Anna, Sohee,
}
pub struct ModeSynthesizeRequest<'a> {
    pub synthesis: SynthesizeRequest,
    pub mode: QtsMode,
    pub instruct: Option<String>,
    pub speaker: Option<PresetSpeaker>,
    pub voice_clone_prompt: Option<&'a VoiceClonePromptV2>,
    pub speaker_embedding: Option<&'a [f32]>,
}

Request mode must match loaded checkpoint mode. Presets use codec embeddings at GGUF-provided IDs. Chinese/Auto preset dialect overrides also come from metadata. VoiceDesign omits speaker embedding entirely. CustomVoice/VoiceDesign synthesis prefills all text plus TTS EOS before codec BOS, matching the official wrappers' non_streaming_mode=true default, including when audio delivery is streamed. Base synthesis retains incremental text/codec prefill. Instruction tokens are projected & prepended separately, preserving assistant/codec alignment:

<|im_start|>user\n{instruct}<|im_end|>\n
<|im_start|>assistant\n{text}<|im_end|>\n<|im_start|>assistant\n

Public additions (Result<T> below means Result<T, Qwen3TtsError>):

QtsMode::as_str(self) -> &'static str
PresetSpeaker::ALL: [PresetSpeaker; 9]
PresetSpeaker::as_str(self) -> &'static str
ModeSynthesizeRequest::validate(&self) -> Result<()>

Qwen3TtsEngine::load_mode(paths: ModeModelPaths, mode: QtsMode) -> Result<Self>
Qwen3TtsEngine::from_model_dir_for_mode(dir: impl AsRef<Path>, mode: QtsMode) -> Result<Self>
Qwen3TtsEngine::mode(&self) -> QtsMode
Qwen3TtsEngine::synthesize_mode(&self, req: &ModeSynthesizeRequest<'_>) -> Result<SynthesizeResult>
Qwen3TtsEngine::synthesize_mode_with_profile(&self, req: &ModeSynthesizeRequest<'_>) -> Result<(SynthesizeResult, SynthesisStageTimings)>
Qwen3TtsEngine::synthesize_mode_streaming<S: StreamingSynthesis + Send>(&self, req: &ModeSynthesizeRequest<'_>, sink: &mut S) -> Result<StreamingSynthesizeResult>

TtsTransformer::load_from_ggufs(talker: &GgufFile, predictor: &GgufFile) -> Result<Self>
TtsTransformer::code_predictor_config(&self) -> &CodePredictorConfig
TtsTransformer::preset_speaker_id(&self, speaker: PresetSpeaker) -> Result<i32>
TtsTransformer::preset_speaker_embedding(&self, speaker: PresetSpeaker, thread_count: usize) -> Result<Vec<f32>>
TtsTransformer::prepend_instruction(&self, inputs: &mut PreparedPrefillInputs, tokens: &[i32], thread_count: usize) -> Result<()>
TextTokenizer::encode_instruction_for_tts(&self, instruct: &str) -> Result<Vec<i32>>
GgufFile::get_str(&self, key: &str) -> Option<String>

CodePredictorConfig exposes hidden_size, n_layers, n_attention_heads, n_key_value_heads, intermediate_size, head_dim, rms_norm_eps, rope_theta & vocab_size. Legacy TtsTransformer::config() remains talker configuration. Predictor embeddings remain talker-wide; projection runs before predictor layers for both prefill & every codebook step. Thus 1.7B uses 2048→1024 while 0.6B uses identity with no projection tensors.

Paths & GGUF export contract

pub struct ModeModelPaths {
    pub talker_gguf: PathBuf,
    pub code_predictor_gguf: PathBuf,
    pub tokenizer_gguf: PathBuf,
    pub vocoder_onnx: PathBuf,
}
ModeModelPaths::from_model_dir(dir: impl AsRef<Path>, mode: QtsMode) -> Self

Default split filenames use mode = base | customvoice | voicedesign:

• qwen3-tts-1.7b-{mode}-talker-f16.gguf
• qwen3-tts-1.7b-{mode}-code-predictor-f16.gguf
• qwen3-tts-1.7b-{mode}-tokenizer.gguf
• qwen3-tts-vocoder.onnx (shared across sizes & modes)

Talker/predictor resolution also accepts q8_0, q6_k, q5_k & q4_k, in that preference order after F16. If a bundled QTS file named qwen3-tts-1.7b-{mode}-{quant}.gguf exists, all three GGUF paths use it. Explicit paths can name any compatible export. Filenames describe local export conventions, not verified downloadable distributions; community GGUFs need QTS tensor names & metadata.

Exports retain existing talker.* & code_pred.* tensor names & tokenizer metadata. New contract:

• Talker width: qwen3-tts.talker.embedding_length (also talker.hidden_size under same namespace, legacy qwen3-tts.embedding_length, or output-norm tensor width).
• Predictor metadata: qwen3-tts.code_pred.{embedding_length,attention.key_length, attention.head_count,attention.head_count_kv,feed_forward_length,layer_count, vocab_size,attention.layer_norm_rms_epsilon,rope.freq_base}. Namespace qwen3-tts.code_predictor.* also works; hidden_size & block_count aliases are accepted. Missing width/head counts/FFN width derive from predictor norm/Q/K/FFN tensors; remaining values inherit legacy talker defaults.
• Optional projection: code_pred.small_to_mtp_projection.weight & .bias. Prefixes talker.code_predictor.small_to_mtp_projection & small_to_mtp_projection also work, in predictor or talker file. GGML shapes must be [talker_width, predictor_width] & [predictor_width]; differing widths require both tensors. F32/F16/quantized weights use normal GGML loading.
• Presets: U32 qwen3-tts.talker.spk_id.{lowercase_speaker}. IDs must fit codec vocabulary; absent requested presets fail explicitly. No numeric preset IDs are built into library.
• Optional dialects: string qwen3-tts.talker.spk_is_dialect.{speaker} plus U32 qwen3-tts.talker.codec_language_id.{dialect}. Boolean false means no dialect.
• Optional model identity strings: qwen3-tts.tts_model_type (base, custom_voice, voice_design) & qwen3-tts.tts_model_size (0.6b, 1.7b). Explicit loader mode applies when identity metadata is absent.

Synthetic tests

These are the crate's unit tests in src/. Run them with rightkit cargo test -p rightkit-qts.

Unit fixtures use tiny local GGUFs & tensors. Coverage includes independent 2048/1024 metadata, 4→2 projection math/bias, cached/recompute equivalence for split/equal widths, separate-file loading, invalid projection shapes, metadata-derived presets/dialects, exact prompt token strings for every mode, instruction-prefill placement, omitted design speaker & request validation. No model downloads are required.

Changes

See CHANGELOG.md.