pub enum Token<'a> {
Word {
text: &'a str,
fixation: usize,
fixation_text: &'a str,
rest_text: &'a str,
},
Separator {
text: &'a str,
},
}Expand description
A piece of the input: a word or a separator.
Tokens borrow from the input. Concatenating Token::text over the
slice returned by tokenize gives the input back unchanged, and for a
word fixation_text + rest_text == text.
Variants§
Word
A word, with its fixation split.
Fields
Separator
Whitespace, punctuation, emoji or other non-word text between words.
Implementations§
Trait Implementations§
impl<'a> Copy for Token<'a>
impl<'a> Eq for Token<'a>
impl<'a> StructuralPartialEq for Token<'a>
Auto Trait Implementations§
impl<'a> Freeze for Token<'a>
impl<'a> RefUnwindSafe for Token<'a>
impl<'a> Send for Token<'a>
impl<'a> Sync for Token<'a>
impl<'a> Unpin for Token<'a>
impl<'a> UnsafeUnpin for Token<'a>
impl<'a> UnwindSafe for Token<'a>
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more