Function batch_encode 

pub fn batch_encode<T: TransformerTokenizer>(
    texts: &[&str],
    tokenizer: &T,
    config: &BatchConfig,
) -> BatchEncoding

Batch-encode multiple texts using a tokenizer.

Each text is independently encoded, then optionally truncated and padded according to the provided BatchConfig.

Padding is added on the right side (standard for most models). For left-padding, use batch_encode_ext.

Arguments

• texts: slice of input strings
• tokenizer: any tokenizer implementing TransformerTokenizer
• config: batch configuration (max_length, padding, truncation, pad token)