Struct KimiTokenizer 

pub struct KimiTokenizer { /* private fields */ }

Implementations

impl KimiTokenizer

pub fn new( encoder: HashMap<Vec<u8>, u32>, special_tokens: HashMap<String, u32>, ) -> Result<Self, KimiTokenizerError>

pub fn vocab_size(&self) -> usize

pub fn special_token_id(&self, name: &str) -> Option<u32>

pub fn encode(&self, text: &str, specials: SpecialTokens) -> Vec<u32>

Encodes text. SpecialTokens::AsText is the real disallowed_special=() path tokenization_kimi.py uses for untrusted/user text: a literal <|...|> substring is BPE-encoded like any other text, never misread as a control token. SpecialTokens::Parse is allowed_special="all": every name in tokenizer_config.json is carved out as its id.

pub fn decode(&self, ids: &[u32]) -> String

Trait Implementations

impl TextTokenizer for KimiTokenizer

fn encode(&self, text: &str, specials: SpecialTokens) -> Vec<usize>

specials is llama.cpp’s parse_special: whether a literal marker in text is the token it names or the characters it is written with. See SpecialTokens for which callers want which.

fn decode(&self, ids: &[usize]) -> String

fn decode_bytes(&self, ids: &[usize]) -> Vec<u8>

The raw bytes, before any UTF-8 decision is made about them.