Struct MtmdAudioGen 

pub struct MtmdAudioGen<'ctx> { /* private fields */ }

Text-to-speech through an mmproj audio-generation pipeline.

This is the other direction of multimodal: where MtmdBitmap feeds audio in, this drives a pipeline that emits it. The loop is explicitly stateless on llama.cpp’s side, so it runs in two phases:

1. Self::set_input, then Self::step_prompt until it returns 0 — the prompt is consumed n_batch tokens at a time.
2. Self::step_gen per frame until it reports stop.
3. Self::output for the finished audio.

Wraps mtmd_helper_gen_audio_*.

Implementations

impl<'ctx> MtmdAudioGen<'ctx>

pub fn new(lctx: &mut LlamaContext<'_>, mctx: &'ctx MtmdContext) -> Result<Self>

Attach a generator to a llama context and an mtmd context.

Errors

Returns MtmdError::ContextCreateFailed if the mmproj has no audio-generation pipeline.

pub fn reset(&mut self)

Clear all state, ready for another utterance.

pub fn set_input( &mut self, request: &MtmdAudioRequest, speaker_ref: Option<&MtmdBitmap>, ) -> Result<()>

Set what to synthesize. speaker_ref is an optional voice reference for pipelines that support cloning.

Errors

Returns MtmdError::EvalError if llama.cpp rejects the request, or MtmdError::InvalidPath if a string contains an interior NUL.

pub fn step_prompt(&mut self, n_batch: i32) -> Result<i32>

Consume up to n_batch prompt tokens.

Returns the number of prompt tokens still outstanding; call again until it returns 0, then move on to Self::step_gen.

Errors

Returns MtmdError::EvalError if llama.cpp reports a negative code.

pub fn step_gen( &mut self, sampled: Option<LlamaToken>, h_state_in: Option<&[f32]>, n_text_embd: usize, ) -> Result<(Option<Vec<f32>>, bool)>

Generate one audio frame.

sampled is the backbone token just sampled, or None for pipelines with no discrete backbone token. h_state_in is the hidden state fed back from the previous step.

Returns (hidden_state, stop). stop marks end-of-speech: the caller must break the loop. The hidden state borrows generator memory that the next step_gen or Self::reset invalidates, hence the &mut self borrow being released before you can call again.

Errors

Returns MtmdError::EvalError on a negative code.

pub fn output(&mut self) -> Result<(i32, Vec<u8>, i64)>

Collect the generated audio.

Returns (sample_rate, bytes, n_samples). bytes is raw PCM or a complete WAV file depending on the request’s MtmdAudioOutType.

Errors

Returns MtmdError::EvalError if nothing has been generated.

Trait Implementations

impl Debug for MtmdAudioGen<'_>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for MtmdAudioGen<'_>

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.