Struct rust_tokenizers::TokenIdsWithOffsets
source · pub struct TokenIdsWithOffsets {
pub ids: Vec<i64>,
pub offsets: Vec<Option<Offset>>,
pub reference_offsets: Vec<Vec<OffsetSize>>,
pub masks: Vec<Mask>,
}
Expand description
Encoded sequence
Intermediate tokenization steps before addition of special tokens, after encoding
Fields§
§ids: Vec<i64>
Vector of token IDs
offsets: Vec<Option<Offset>>
Offset information (as start and end positions) in relation to the original text. Tokens that can not be related to the original source are registered as None.
reference_offsets: Vec<Vec<OffsetSize>>
Offset information (as a sequence of positions) in relation to the original text. Tokens that can not be related to the original source are registered as None.
masks: Vec<Mask>
Masks tokens providing information on the type of tokens. This vector has the same length as token_ids.
Trait Implementations§
source§impl Clone for TokenIdsWithOffsets
impl Clone for TokenIdsWithOffsets
source§fn clone(&self) -> TokenIdsWithOffsets
fn clone(&self) -> TokenIdsWithOffsets
Returns a copy of the value. Read more
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source
. Read moresource§impl Debug for TokenIdsWithOffsets
impl Debug for TokenIdsWithOffsets
source§impl PartialEq for TokenIdsWithOffsets
impl PartialEq for TokenIdsWithOffsets
source§fn eq(&self, other: &TokenIdsWithOffsets) -> bool
fn eq(&self, other: &TokenIdsWithOffsets) -> bool
This method tests for
self
and other
values to be equal, and is used
by ==
.impl Eq for TokenIdsWithOffsets
impl StructuralEq for TokenIdsWithOffsets
impl StructuralPartialEq for TokenIdsWithOffsets
Auto Trait Implementations§
impl RefUnwindSafe for TokenIdsWithOffsets
impl Send for TokenIdsWithOffsets
impl Sync for TokenIdsWithOffsets
impl Unpin for TokenIdsWithOffsets
impl UnwindSafe for TokenIdsWithOffsets
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more