Trait Tokenizer 

pub trait Tokenizer {
    // Required methods
    fn cls_id(&self) -> u32;
    fn sep_id(&self) -> u32;
    fn mask_id(&self) -> u32;
    fn pad_id(&self) -> u32;
    fn mask_token(&self) -> &str;
    fn encode(&self, text: &str) -> Vec<u32>;
}

Abstraction over a checkpoint tokenizer. Mirrors TokenizerLike in laya-ts. encode corresponds to HF tokenizer(text, add_special_tokens=False)["input_ids"].

Required Methods

fn cls_id(&self) -> u32

fn sep_id(&self) -> u32

fn mask_id(&self) -> u32

fn pad_id(&self) -> u32

fn mask_token(&self) -> &str

fn encode(&self, text: &str) -> Vec<u32>

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl Tokenizer for LayaTokenizer