Struct rust_tokenizers::TokenizedInput

pub struct TokenizedInput {
    pub token_ids: Vec<i64>,
    pub segment_ids: Vec<i8>,
    pub special_tokens_mask: Vec<i8>,
    pub overflowing_tokens: Vec<i64>,
    pub num_truncated_tokens: usize,
    pub token_offsets: Vec<Option<Offset>>,
    pub reference_offsets: Vec<Vec<OffsetSize>>,
    pub mask: Vec<Mask>,
}

Tokenized Input, ready for processing in language models

This represents the final output of the encoding process (tokenized sentence with encoded values)

Fields

token_ids: Vec<i64>

Vector of token IDs

segment_ids: Vec<i8>

Vector segments ids (for example for BERT segments are separated with a [SEP] marker, each incrementing the segment ID). This vector has the same length as token_ids.

special_tokens_mask: Vec<i8>

Flags tokens as special tokens (1) or not (0). This vector has the same length as token_ids.

overflowing_tokens: Vec<i64>

Vector containing overflowing tokens, populated following a truncation step

num_truncated_tokens: usize

Number of overflowing tokens following a truncation step. this equals the length overflowing_tokens

token_offsets: Vec<Option<Offset>>

Offset information (as start and end positions) in relation to the original text. Tokens that can not be related to the original source are registered as None.

reference_offsets: Vec<Vec<OffsetSize>>

Offset information (as a sequence of positions) in relation to the original text. Tokens that can not be related to the original source are registered as None.

mask: Vec<Mask>

Masks tokens providing information on the type of tokens. This vector has the same length as token_ids.

Trait Implementations

impl Clone for TokenizedInput

fn clone(&self) -> TokenizedInput

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizedInput

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq<TokenizedInput> for TokenizedInput

fn eq(&self, other: &TokenizedInput) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd<TokenizedInput> for TokenizedInput

fn partial_cmp(&self, other: &TokenizedInput) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

This method tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

This method tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

This method tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

This method tests greater than or equal to (for self and other) and is used by the >= operator.

impl Eq for TokenizedInput

impl StructuralEq for TokenizedInput

impl StructuralPartialEq for TokenizedInput