pub enum TokenFlags {
WordBoundary,
SentenceEnd,
}Expand description
Flags for what type of token is being returned.
Variants§
WordBoundary
If set, this token marks the start of a new word.
SentenceEnd
If set, this token marks the end of a sentence, meaning the token is equal to “.”, “!”, or “?”. Some models may not have this token.
Trait Implementations§
Source§impl Debug for TokenFlags
impl Debug for TokenFlags
Auto Trait Implementations§
impl Freeze for TokenFlags
impl RefUnwindSafe for TokenFlags
impl Send for TokenFlags
impl Sync for TokenFlags
impl Unpin for TokenFlags
impl UnwindSafe for TokenFlags
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more