Struct BertTokenizer 

pub struct BertTokenizer { /* private fields */ }

BERT-style tokenizer combining basic tokenization and WordPiece subword segmentation.

Special tokens:

• [CLS] (classification): prepended to every encoded sequence
• [SEP] (separator): appended after each segment
• [MASK] (masking): placeholder for masked-language-model pre-training
• [PAD] (padding): used to fill sequences to a target length
• [UNK] (unknown): substituted for tokens not present in the vocabulary

Example

use std::collections::HashMap;
use scirs2_text::tokenizers::bert::BertTokenizer;

let mut vocab: HashMap<String, u32> = HashMap::new();
for (i, tok) in ["[PAD]","[UNK]","[CLS]","[SEP]","[MASK]",
                  "hello","world","##ing","play","##ed"].iter().enumerate() {
    vocab.insert(tok.to_string(), i as u32);
}
let tokenizer = BertTokenizer::new(vocab, true);
let ids = tokenizer.encode("Hello World").unwrap();
assert_eq!(ids[0], tokenizer.cls_token_id());

Implementations

impl BertTokenizer

pub fn new(vocab: HashMap<String, u32>, lowercase: bool) -> Self

Build a BertTokenizer from a token → id vocabulary map.

All five special tokens ([PAD], [UNK], [CLS], [SEP], [MASK]) are inserted into the vocabulary if absent.

pub fn from_vocab_file(path: &str) -> Result<Self>

Load a tokenizer from a vocab.txt file (one token per line; line index = token ID, 0-based).

Returns an error if the file cannot be read or if the resulting vocabulary is missing required special tokens after auto-insertion.

pub fn with_max_len(self, max_len: usize) -> Self

Override the maximum sequence length (default 512).

pub fn cls_token_id(&self) -> u32

Returns the [CLS] token ID.

pub fn sep_token_id(&self) -> u32

Returns the [SEP] token ID.

pub fn pad_token_id(&self) -> u32

Returns the [PAD] token ID.

pub fn mask_token_id(&self) -> u32

Returns the [MASK] token ID.

pub fn unk_token_id(&self) -> u32

Returns the [UNK] token ID.

pub fn vocab_size(&self) -> usize

Vocabulary size.

pub fn vocab(&self) -> &HashMap<String, u32>

Return a reference to the full token → id vocabulary map.

pub fn lowercase(&self) -> bool

Return whether this tokenizer lowercases input text.

pub fn tokenize(&self, text: &str) -> Vec<String>

Tokenize text into a list of subword strings.

Applies basic tokenization (whitespace + punctuation split, optional lowercasing) followed by WordPiece subword segmentation. Unknown characters/words map to "[UNK]".

pub fn encode(&self, text: &str) -> Result<Vec<u32>>

Encode a single text segment as [CLS] tokens [SEP].

Returns the flat sequence of token IDs. Use encode_pair for two-segment inputs (e.g. question + context).

pub fn encode_pair( &self, text_a: &str, text_b: &str, ) -> Result<(Vec<u32>, Vec<u32>)>

Encode a pair of text segments (e.g. sentence A and sentence B).

Layout: [CLS] A-tokens [SEP] B-tokens [SEP]

Returns (token_ids, token_type_ids) where token_type_ids[i] is 0 for the first segment and 1 for the second.

pub fn encode_single( &self, text: &str, max_length: usize, padding: bool, truncation: bool, ) -> Result<BertEncoding>

Build a single BertEncoding for text, with optional padding and truncation to max_length.

If padding is true, short sequences are padded with [PAD] to reach max_length. If truncation is true, long sequences are trimmed (preserving [CLS] and [SEP]).

pub fn encode_batch( &self, texts: &[&str], max_length: usize, padding: bool, truncation: bool, ) -> Result<BatchEncoding>

Encode a batch of texts with consistent sequence length.

When padding is true, all sequences in the batch are padded to the longest (or to max_length, whichever is smaller). When truncation is true, sequences exceeding max_length are truncated.

pub fn decode(&self, ids: &[u32]) -> String

Decode a sequence of token IDs back to a human-readable string.

Special tokens ([CLS], [SEP], [PAD], [MASK]) are skipped. WordPiece continuation tokens (prefixed with ##) are merged directly onto the preceding piece without a space.

pub fn convert_token_to_id(&self, token: &str) -> Option<u32>

Convert token string to its ID (exposed for testing / downstream use).

pub fn convert_id_to_token(&self, id: u32) -> Option<&str>

Convert token ID to its string representation.

Trait Implementations

impl Clone for BertTokenizer

fn clone(&self) -> BertTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BertTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.