Struct BertTokenizer 

pub struct BertTokenizer { /* private fields */ }

Wrapper around HuggingFace tokenizer configured for BERT-style encoding.

Implementations

impl BertTokenizer

pub fn from_dir(dir: &Path, max_length: usize) -> Result<BertTokenizer, Error>

Load tokenizer from a model directory containing:

• tokenizer.json
• config.json
• special_tokens_map.json
• tokenizer_config.json

pub fn from_bytes( tokenizer_json: &[u8], config_json: &[u8], special_tokens_map_json: &[u8], tokenizer_config_json: &[u8], max_length: usize, ) -> Result<BertTokenizer, Error>

Load tokenizer from raw file bytes.

pub fn encode_batch(&self, texts: &[&str]) -> Result<TokenizedBatch, Error>

Tokenize a batch of texts.

Returns input_ids, attention_mask, and token_type_ids for each text, all padded to the same length (longest in batch).