Struct Tokenizer 

pub struct Tokenizer {
    pub cls_token: String,
    pub sep_token: String,
    pub pad_token: String,
    pub mask_token: String,
    pub cls_token_id: u32,
    pub sep_token_id: u32,
    pub pad_token_id: u32,
    pub mask_token_id: u32,
    /* private fields */
}

A loaded tokenizer plus the four special tokens upstream requires.

Fields

cls_token: String

sep_token: String

pad_token: String

mask_token: String

cls_token_id: u32

sep_token_id: u32

pad_token_id: u32

mask_token_id: u32

Implementations

impl Tokenizer

pub fn from_dir(dir: impl AsRef<Path>) -> Result<Tokenizer, Error>

Load tokenizer.json and tokenizer_config.json from a directory.

pub fn encode(&self, text: &str) -> Result<Vec<u32>, Error>

Encode without special tokens, as every upstream call site does.

pub fn vocab_size(&self) -> usize

Trait Implementations

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.