Struct TtsModelExecutor 

pub struct TtsModelExecutor { /* private fields */ }

Qwen3-TTS executor: text-to-speech synthesis.

Implementations

impl TtsModelExecutor

pub fn from_path( model_path: &str, device: CandleDevice, dtype: DType, ) -> Result<Self>

Load from model directory containing:

• config.json (TalkerConfig)
• model.safetensors (Talker weights)
• speech_tokenizer/model.safetensors (Vocoder weights)
• tokenizer_config.json + vocab.json + merges.txt (text tokenizer)

pub fn synthesize(&mut self, text: &str, language: &str) -> Result<Vec<f32>>

Synthesize speech from text.

Returns PCM samples at 24kHz as Vec.

Prompt structure (matches Python/qwen3-tts-rs): Prefill: [role_prefix(3)] + [tts_text_prefix(6) + codec_prefix(6)] + [first_text + codec_bos] Trailing: text_projection(remaining_text + tts_eos) — added per decode step

pub fn synthesize_streaming<F: FnMut(usize, &[f32])>( &mut self, text: &str, language: &str, chunk_frames: usize, on_chunk: F, ) -> Result<Vec<Vec<f32>>>

Streaming TTS: calls on_chunk with each audio chunk as soon as it’s ready.

Each chunk is chunk_frames codec frames decoded to audio (~800ms at default 10 frames). First chunk arrives after chunk_frames decode steps (~2-3s for 0.6B).

pub fn sample_rate(&self) -> usize

Get the output sample rate.

pub fn config(&self) -> &TalkerConfig

pub fn synthesize_voice_clone( &mut self, text: &str, language: &str, ref_audio_path: &str, ref_text: &str, ) -> Result<Vec<f32>>

Synthesize speech with voice cloning from a reference audio.

Uses ICL (in-context learning) prompting: the reference audio is encoded to codec tokens and prepended to the generation prompt, along with a speaker embedding extracted via ECAPA-TDNN.

Returns PCM samples at 24kHz as Vec.

Trait Implementations

impl ModelExecutor for TtsModelExecutor

fn info(&self) -> &ModelInfo

Get model information and metadata

fn prefill<'life0, 'life1, 'async_trait>( &'life0 self, _input: &'life1 PrefillInput, ) -> Pin<Box<dyn Future<Output = Result<PrefillOutput>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Execute prefill phase (process initial prompt)

fn decode<'life0, 'life1, 'async_trait>( &'life0 self, _input: &'life1 DecodeInput, ) -> Pin<Box<dyn Future<Output = Result<DecodeOutput>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Execute decode phase (generate next token)

fn capabilities(&self) -> ExecutorCapabilities

Get executor capabilities

fn release_cache(&self, _: &str)

Release KV cache and state without asserting successful completion.

fn status(&self) -> ExecutorStatus

Get current executor status

fn execution_resource_authority(&self) -> ExecutionResourceAuthority

Selects the single authority for request-lifetime model resources. Existing executors remain on the transitional legacy-engine path by default. A runtime that returns PlanRuntime must return the shared runtime’s opaque cache handle from prefill/decode and delegate release of that authority from release_cache.

fn admission_limits( &self, ) -> Result<Option<ExecutorAdmissionLimits>, FerrumError>

Immutable admission limits compiled into this executor’s runtime plan. A PlanRuntime executor must return Some; legacy executors may defer to the engine-owned scheduler and recurrent-state limits.

fn resolved_model_plan(&self) -> Option<&ResolvedModelPlan>

Returns the immutable product plan that owns planning, provider selection, and resource authority for a plan-runtime executor. Legacy executors return None; PlanRuntime executors must expose the exact plan used for provisioning and dispatch.

fn plan_runtime_resource_snapshot( &self, ) -> Result<Option<PlanRuntimeResourceSnapshot>, FerrumError>

Returns the authoritative memory breakdown for a shared plan runtime. LegacyEngine executors return None; PlanRuntime executors must return Some while they are ready.

fn supports_native_unified_decode(&self) -> bool

Whether this executor’s backend can run the unified mixed prefill+decode forward natively. When false, the engine routes Qwen3-MoE batches through the legacy split path. Reported by the (backend-aware) executor so the engine stays backend-agnostic — replaces a cfg(target_os) branch that previously hard-coded “Metal/CPU lack native unified” in the hot path.

fn kv_capacity(&self) -> Option<usize>

Per-request KV capacity in tokens when the executor owns a smaller runtime cache window than the model’s declared context length.

fn attach_execution_event_sink(&self, _sink: Arc<dyn ExecutionEventSink>)

Installs the product-owned execution event sink before requests start.

fn execution_capacity_epochs( &self, ) -> Result<Option<ExecutorAdmissionEpochs>, FerrumError>

Current plan-local capacity evidence for scheduler wake suppression. Legacy-engine executors return None; an executor declaring ExecutionResourceAuthority::PlanRuntime must return Some.

fn write_execution_capacity_snapshot( &self, availability: &mut Vec<CapacityAvailabilityEpoch>, ) -> Result<Option<ExecutorAdmissionEpochs>, FerrumError>

Writes the canonical per-source availability generations into caller-owned storage and returns the matching global audit epochs. Executors with typed dynamic admission override this to avoid allocating on steady scheduler ticks.

fn register_execution_capacity_waiter( &self, _observed: &CapacityWaitCondition, ) -> Result<Option<ExecutorCapacityWaitRegistration>, FerrumError>

Synchronously subscribes to every source named by one passive capacity wait. The returned registration must remain alive until it is awaited or deliberately cancelled by being dropped.

fn try_admit_prefill( &self, _input: ExecutorPrefillAdmission<'_>, ) -> Result<ExecutorPrefillAdmissionDecision, FerrumError>

Probe and retain the exact request/sequence authority needed by a future prefill. No provider encode, kernel launch, or device submit may occur in this method.

fn cancel_prefill_admission(&self, _request_id: &RequestId) -> bool

Release an admitted but not yet active prefill authority.

fn write_execution_capacity_release_sources( &self, _preemption: &ExecutorExecutionCapacityPreemption, sources: &mut Vec<CapacityAvailabilitySource>, ) -> Result<bool, FerrumError>

Writes the exact availability sources advanced when this request authority is preempted for recompute.

fn preempt_execution_capacity<'life0, 'async_trait>( &'life0 self, _preemption: ExecutorExecutionCapacityPreemption, ) -> Pin<Box<dyn Future<Output = Result<ExecutorExecutionCapacityPreemptionReceipt, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Retire one exact request-scoped runtime authority for recompute.

fn maintain_prefill_backing( &self, _request_id: &RequestId, ) -> Result<ExecutorPrefillMaintenanceOutcome, FerrumError>

Consume one retained logical/physical backing deferral after the scheduler waiting lock has been released. Implementations must perform at most one bounded maintenance attempt and publish capacity epochs only after real backing is installed.

fn reserve_kv_slots( &self, _requests: &[KvSlotRequest], ) -> Result<Option<KvSlotReservation>, FerrumError>

Reserve model-owned KV slots before a forward is dispatched.

fn kv_slot_capacity_snapshot(&self) -> Option<KvSlotCapacitySnapshot>

Snapshot model-owned paged-KV capacity without allocating slots.

fn recurrent_state_spec( &self, _request_id: &RequestId, _input_tokens: &[TokenId], ) -> Result<Option<RecurrentStateSpec>, FerrumError>

Recurrent-state allocation spec for this request, when the model has state-space or hybrid layers that need per-request recurrent state.

fn prefill_with_capacity<'life0, 'life1, 'async_trait>( &'life0 self, input: &'life1 PrefillInput, ) -> Pin<Box<dyn Future<Output = Result<ExecutorPrefillOutcome, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Execute one exact prefill chunk with an explicit pre-submit capacity deferral edge. Legacy executors inherit full-prefill behavior.

fn batch_prefill<'life0, 'life1, 'async_trait>( &'life0 self, inputs: &'life1 [PrefillInput], ) -> Pin<Box<dyn Future<Output = Result<Vec<PrefillOutput>, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Batch prefill: process multiple prompts’ prefill in ONE forward pass.

fn batch_prefill_with_capacity<'life0, 'life1, 'async_trait>( &'life0 self, _inputs: &'life1 [PrefillInput], ) -> Pin<Box<dyn Future<Output = Result<ExecutorBatchPrefillOutcome, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Attempt one physical, capacity-aware prefill batch.

fn plan_runtime_prefill_with_capacity<'life0, 'life1, 'async_trait>( &'life0 self, _input: &'life1 PlanRuntimePrefillInput, ) -> Pin<Box<dyn Future<Output = Result<PlanRuntimePrefillOutcome, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Tensor-free prefill for executors that declare ExecutionResourceAuthority::PlanRuntime.

fn plan_runtime_batch_prefill_with_capacity<'life0, 'life1, 'async_trait>( &'life0 self, _inputs: &'life1 [PlanRuntimePrefillInput], ) -> Pin<Box<dyn Future<Output = Result<PlanRuntimeBatchPrefillOutcome, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Attempt one physical tensor-free prefill batch.

fn discard_plan_runtime_prefill( &self, authority: PlanRuntimePrefillAuthority, ) -> Result<(), FerrumError>

Discard an exact prefill authority after engine-side validation, sampling, or scheduler commit fails.

fn batch_decode<'life0, 'life1, 'async_trait>( &'life0 self, inputs: &'life1 [DecodeInput], ) -> Pin<Box<dyn Future<Output = Result<Vec<DecodeOutput>, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Batch decode: process multiple sequences in one forward pass.

fn batch_decode_with_capacity<'life0, 'life1, 'async_trait>( &'life0 self, inputs: &'life1 [DecodeInput], ) -> Pin<Box<dyn Future<Output = Result<ExecutorBatchDecodeOutcome, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Batch decode with an explicit pre-submit capacity deferral edge.

fn plan_runtime_batch_decode_with_capacity<'life0, 'life1, 'async_trait>( &'life0 self, _inputs: &'life1 [PlanRuntimeDecodeInput], ) -> Pin<Box<dyn Future<Output = Result<PlanRuntimeBatchDecodeOutcome, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Tensor-free batch decode for executors that declare ExecutionResourceAuthority::PlanRuntime.

fn unified_decode<'life0, 'life1, 'async_trait>( &'life0 self, _batch: &'life1 UnifiedBatch, ) -> Pin<Box<dyn Future<Output = Result<Vec<Option<Vec<f32>>>, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Unified mixed-batch forward: process a UnifiedBatch containing any combination of prefill chunks (one or more q_tokens per item, possibly continuing from pos_offset > 0) and decode steps (q_tokens.len() == 1, is_final_chunk = true) in a single model forward pass.

fn forward<'life0, 'life1, 'async_trait>( &'life0 self, _input: &'life1 Arc<dyn TensorLike>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn TensorLike>, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Optional: full forward pass (for non-autoregressive use cases)

fn truncate_kv<'life0, 'life1, 'async_trait>( &'life0 self, _kv_cache: &'life1 Arc<dyn KvCacheHandle>, _new_len: usize, ) -> Pin<Box<dyn Future<Output = Result<(), FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Roll the KV cache for this executor’s sequence back to new_len. Used by speculative decoding on partial rejection so the next iteration sees a KV prefix that matches the accepted token stream. Default: Ok(()) — executors that don’t cache per-sequence state (stub, mock) are inherently tolerant; real LLM executors override.

fn forward_verify<'life0, 'life1, 'async_trait>( &'life0 self, inputs: &'life1 [DecodeInput], ) -> Pin<Box<dyn Future<Output = Result<Vec<DecodeOutput>, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Multi-position decode-verify: one forward over N+1 tokens, producing one logits row per position. Used by speculative decoding’s target path so we don’t pay N+1 sequential forwards.

fn cache_metrics_snapshot(&self) -> Option<Value>

Optional model/executor cache metrics.

fn lora_metrics_snapshot(&self) -> Option<Value>

Optional LoRA runtime metrics.

fn prepare_startup<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<(), FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Complete executor-owned startup preparation before either product entrypoint can accept a request.

fn warmup<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<(), FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Warm up executor (load model, allocate memory, etc.)

fn shutdown<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<(), FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Shutdown executor gracefully

fn complete_cache( &self, completion: ExecutorSequenceCompletion, ) -> Result<(), FerrumError>

Complete and release one cache authority with product-authoritative terminal token counts.