Struct rust_tokenizers::TokenIdsWithOffsets [−][src]
pub struct TokenIdsWithOffsets { pub ids: Vec<i64>, pub offsets: Vec<Option<Offset>>, pub reference_offsets: Vec<Vec<OffsetSize>>, pub masks: Vec<Mask>, }
Expand description
Encoded sequence
Intermediate tokenization steps before addition of special tokens, after encoding
Fields
ids: Vec<i64>
Vector of token IDs
offsets: Vec<Option<Offset>>
Offset information (as start and end positions) in relation to the original text. Tokens that can not be related to the original source are registered as None.
reference_offsets: Vec<Vec<OffsetSize>>
Offset information (as a sequence of positions) in relation to the original text. Tokens that can not be related to the original source are registered as None.
masks: Vec<Mask>
Masks tokens providing information on the type of tokens. This vector has the same length as token_ids.
Trait Implementations
This method tests for self
and other
values to be equal, and is used
by ==
. Read more
This method tests for !=
.
Auto Trait Implementations
impl RefUnwindSafe for TokenIdsWithOffsets
impl Send for TokenIdsWithOffsets
impl Sync for TokenIdsWithOffsets
impl Unpin for TokenIdsWithOffsets
impl UnwindSafe for TokenIdsWithOffsets
Blanket Implementations
Mutably borrows from an owned value. Read more