Struct rust_tokenizers::Token
source · pub struct Token {
pub text: String,
pub offset: Offset,
pub reference_offsets: Vec<OffsetSize>,
pub mask: Mask,
}
Expand description
Owned token that references the original text but stores its own string representation.
Fields§
§text: String
String representation
offset: Offset
Start and end positions of the token with respect to the original text
reference_offsets: Vec<OffsetSize>
Sequence of positions with respect to the original text contained in the token.
For example, if the token offset is start: 4, end: 10
, corresponding reference_offsets are [4, 5, 6, 7, 8, 9]
mask: Mask
Mask indicating the type of the token
Implementations§
Trait Implementations§
source§impl ConsolidatableTokens<Token> for Vec<Token>
impl ConsolidatableTokens<Token> for Vec<Token>
source§fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, Token> ⓘ
fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, Token> ⓘ
Creates an iterator from a sequence of
ConsolidatableTokens
.source§impl PartialEq for Token
impl PartialEq for Token
source§impl TokenTrait for Token
impl TokenTrait for Token
impl Eq for Token
impl StructuralEq for Token
impl StructuralPartialEq for Token
Auto Trait Implementations§
impl RefUnwindSafe for Token
impl Send for Token
impl Sync for Token
impl Unpin for Token
impl UnwindSafe for Token
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more