pub trait TokenizerEncoder: Send + Sync {
// Required method
fn encode_ids(
&self,
text: &str,
special_ranges: &[Range<usize>],
) -> Result<Vec<u32>, String>;
}Expand description
An encoder that maps prompt text to model token IDs.
Implemented for PromptTokenizer; other tokenizers can implement it to
provide token encoding for a prompt encoding.
Required Methods§
Sourcefn encode_ids(
&self,
text: &str,
special_ranges: &[Range<usize>],
) -> Result<Vec<u32>, String>
fn encode_ids( &self, text: &str, special_ranges: &[Range<usize>], ) -> Result<Vec<u32>, String>
Encode text into token IDs.
special_ranges are the byte ranges of text that hold explicit
special tokens. The caller must pass ranges in ascending order that do
not overlap, lie within text, and start and end on char boundaries;
RenderedPrompt::special_ranges meets these conditions.
The implementation must encode each range as the token ID of its spelling and the plain text between the ranges without the added vocabulary, so a special-token spelling in plain text encodes as the token IDs of its characters. The implementation must add no tokens, such as a BOS token, beyond those that the text encodes to.
§Errors
Returns an error message when the implementation cannot encode text.
Dyn Compatibility§
This trait is dyn compatible.
In older versions of Rust, dyn compatibility was called "object safety".