Struct PromptTokenizer 

pub struct PromptTokenizer { /* private fields */ }

A tokenizer prepared for prompt encoding.

It holds a copy of the tokenizer without its added vocabulary, which encodes plain text, and the added vocabulary, which supplies special token IDs.

Implementations

impl PromptTokenizer

pub fn new(tokenizer: &Tokenizer) -> Self

Prepare tokenizer for prompt encoding. This clones the tokenizer once.

Trait Implementations

impl Clone for PromptTokenizer

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl TokenizerEncoder for PromptTokenizer

fn encode_ids( &self, text: &str, special_ranges: &[Range<usize>], ) -> Result<Vec<u32>, String>

Encode the plain text regions as one batch with the tokenizer without its added vocabulary, and each special range as the added-vocabulary token ID of its spelling. The batch runs in parallel unless the TOKENIZERS_PARALLELISM environment variable disables it. The tokenizer adds no tokens, such as a BOS token, beyond those that the text encodes to, provided that it has no padding configured.

Errors

Returns an error message when the tokenizer fails to encode a plain text region or the spelling of a special range is not in the added vocabulary.

Panics

Panics if a range starts before the end of the previous range, ends before it starts, extends past the end of text, or does not start and end on char boundaries.