Struct TokenizerWrapper 

pub struct TokenizerWrapper { /* private fields */ }

Thread-safe tokenizer wrapper

Thread Safety

This wrapper uses Arc<Tokenizer> for thread-safe access. Multiple threads can encode text concurrently using the same tokenizer.

Performance

• Tokenization: ~0.1-0.5ms per input (100-500 tokens)
• Thread-safe without locks (immutable after creation)
• Batch encoding is more efficient than individual calls

Example

let tokenizer = Arc::new(
    TokenizerWrapper::from_pretrained(
        "microsoft/deberta-v3-base",
        TokenizerConfig::default(),
    )?
);

let handles: Vec<_> = (0..4)
    .map(|i| {
        let tok = Arc::clone(&tokenizer);
        thread::spawn(move || tok.encode(&format!("Text {}", i)))
    })
    .collect();

for handle in handles {
    let encoding = handle.join().unwrap()?;
    println!("Encoded {} tokens", encoding.len());
}

Implementations

impl TokenizerWrapper

pub fn from_pretrained( model_name: &str, config: TokenizerConfig, ) -> Result<Self>

Load a tokenizer from HuggingFace Hub

Arguments

• model_name - HuggingFace model identifier (e.g., “microsoft/deberta-v3-base”)
• config - Tokenizer configuration

Supported Models

• DeBERTa: microsoft/deberta-v3-base (PromptInjection)
• RoBERTa: roberta-base (Toxicity, Sentiment)
• BERT: bert-base-uncased
• Any HuggingFace model with a tokenizer

Example

let tokenizer = TokenizerWrapper::from_pretrained(
    "microsoft/deberta-v3-base",
    TokenizerConfig::default(),
)?;

pub fn encode(&self, text: &str) -> Result<Encoding>

Encode a single text string

Arguments

• text - Input text to tokenize

Returns

Encoding with token IDs and attention mask

Example

let encoding = tokenizer.encode("Hello, world!")?;
println!("Token IDs: {:?}", encoding.input_ids);
println!("Attention mask: {:?}", encoding.attention_mask);

pub fn encode_batch(&self, texts: &[&str]) -> Result<Vec<Encoding>>

Encode multiple texts in batch

Batch encoding is more efficient than encoding texts individually.

Arguments

• texts - Slice of text strings

Returns

Vector of Encoding results (one per input text)

Example

let texts = vec!["First text", "Second text", "Third text"];
let encodings = tokenizer.encode_batch(&texts)?;

assert_eq!(encodings.len(), 3);
for encoding in encodings {
    println!("Length: {}", encoding.len());
}

pub fn config(&self) -> &TokenizerConfig

Get the tokenizer configuration

pub fn vocab_size(&self) -> usize

Get the vocabulary size

Returns the size of the tokenizer’s vocabulary.

Trait Implementations

impl Clone for TokenizerWrapper

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.