Struct rust_tokenizers::TokensWithOffsets
source · pub struct TokensWithOffsets {
pub tokens: Vec<String>,
pub offsets: Vec<Option<Offset>>,
pub reference_offsets: Vec<Vec<OffsetSize>>,
pub masks: Vec<Mask>,
}
Expand description
Tokenized sequence
Intermediate tokenization steps before encoding, addition of special tokens and truncation
Fields§
§tokens: Vec<String>
Vector of token strings
offsets: Vec<Option<Offset>>
Offset information (as start and end positions) in relation to the original text. Tokens that can not be related to the original source are registered as None.
reference_offsets: Vec<Vec<OffsetSize>>
Offset information (as a sequence of positions) in relation to the original text. Tokens that can not be related to the original source are registered as None.
masks: Vec<Mask>
Masks tokens providing information on the type of tokens. This vector has the same length as token_ids.
Trait Implementations§
source§impl Clone for TokensWithOffsets
impl Clone for TokensWithOffsets
source§fn clone(&self) -> TokensWithOffsets
fn clone(&self) -> TokensWithOffsets
Returns a copy of the value. Read more
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source
. Read moreAuto Trait Implementations§
impl RefUnwindSafe for TokensWithOffsets
impl Send for TokensWithOffsets
impl Sync for TokensWithOffsets
impl Unpin for TokensWithOffsets
impl UnwindSafe for TokensWithOffsets
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more