Struct rust_tokenizers::Token [−][src]
pub struct Token { pub text: String, pub offset: Offset, pub reference_offsets: Vec<OffsetSize>, pub mask: Mask, }
Expand description
Owned token that references the original text but stores its own string representation.
Fields
text: String
String representation
offset: Offset
Start and end positions of the token with respect to the original text
reference_offsets: Vec<OffsetSize>
Sequence of positions with respect to the original text contained in the token.
For example, if the token offset is start: 4, end: 10
, corresponding reference_offsets are [4, 5, 6, 7, 8, 9]
mask: Mask
Mask indicating the type of the token
Implementations
Creates a new owned token from a String
.
Parameters
- text (
String
): text reference
Example
use rust_tokenizers::Token; let text = "world".to_string(); let token = Token::new(text);
Trait Implementations
fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, Token>ⓘNotable traits for ConsolidatedTokenIterator<'a, T>
impl<'a, T> Iterator for ConsolidatedTokenIterator<'a, T> where
T: TokenTrait, type Item = &'a [T];
[src]
fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, Token>ⓘNotable traits for ConsolidatedTokenIterator<'a, T>
impl<'a, T> Iterator for ConsolidatedTokenIterator<'a, T> where
T: TokenTrait, type Item = &'a [T];
[src]Creates an iterator from a sequence of ConsolidatableTokens
.
Auto Trait Implementations
impl RefUnwindSafe for Token
impl UnwindSafe for Token
Blanket Implementations
Mutably borrows from an owned value. Read more