Struct MoshiTokenizer 

pub struct MoshiTokenizer {
    pub text_start_token: u32,
    pub text_pad_token: u32,
    /* private fields */
}

Moshi text tokenizer (SentencePiece, 32k vocab).

Fields

text_start_token: u32

text_pad_token: u32

Implementations

impl MoshiTokenizer

pub fn open(path: impl AsRef<Path>) -> Result<MoshiTokenizer, Error>

pub fn encode(&self, text: &str) -> Result<Vec<u32>, Error>

pub fn decode_piece(&self, id: u32) -> Result<String, Error>

pub fn prompt_frame_tokens( &self, prompt: &str, num_frames: usize, ) -> Result<Vec<u32>, Error>

Build per-frame text tokens for one-way generation: pad + inner-monologue prefix.