Struct Encoded 

pub struct Encoded<'a> { /* private fields */ }

Output produced by a Tokenizer::encode implementation.

Use Encoded::token_ids to get the token IDs to feed to a model, and Encoded::text_for_token_range to map token ID ranges back to the corresponding input text.

Implementations

impl<'a> Encoded<'a>

pub fn token_ids(&self) -> &[TokenId]

Return the sequence of token IDs that the input was tokenized into.

pub fn into_token_ids(self) -> Vec<TokenId>

Consume self and return a list of token IDs.

This is a convenient way to discard other information from the encoded output and get the token IDs as an owned vector.

pub fn token_offsets(&self) -> &[usize]

Return the byte offsets of the start of each token in the input sequence. If the input contained two sequences, the offsets are assigned as if the two sequences were concatenated.

pub fn token_type_ids(&self) -> impl Iterator<Item = usize>

Return an iterator of the inputs for the token_type_ids input field in the model, if it has one.

pub fn text_for_token_range(&self, range: Range<usize>) -> Option<&'a str>

Return the text from the input sequence(s) that corresponds to a range of token indices. If the input contained two sequences, the range must lie entirely within one of them.

Trait Implementations

impl<'a> Debug for Encoded<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.