Enum rust_tokenizers::Mask [−][src]
Variants
The token has no particular mask. This is the default situation. It may indicate that further processing can be done on a token.
The token represents a whitespace (in any shape or form)
The token represents punctuation (in any shape or form)
The token represents a single Chinese/Japanese/Korean character (including kana and hangul)
The token is a special marker (such as a separator marker, a class marker, etc)
The token is the begin in a series of subtokens, the offset refers specifically to the sub-token. Subsequent tokens in this sequence will carry the ‘Continuation’ mask
The token is the continuation of the previous token, the offset refers specifically to the sub-token. All but the first sub-token in a sequence carry this mask (the first carries ‘Begin’). (this is the reverse of Mask::Unfinished)
The token is the start of a token but not finished yet. All but the last sub-token in the a token sequence carry this mask. This is the reverse of Mask::Continuation.
The token is out of vocabulary, it is unknown by the tokenizer and it will decode to unknown. Tokens that can be decoded properly (but may still be out of vocabulary) should not set this.
Trait Implementations
impl Clone for Mask
[src]
impl Copy for Mask
[src]
impl Debug for Mask
[src]
impl Default for Mask
[src]
impl<'de> Deserialize<'de> for Mask
[src]
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
[src]
__D: Deserializer<'de>,
impl PartialEq<Mask> for Mask
[src]
impl PartialOrd<Mask> for Mask
[src]
fn partial_cmp(&self, other: &Mask) -> Option<Ordering>
[src]
#[must_use]pub fn lt(&self, other: &Rhs) -> bool
1.0.0[src]
#[must_use]pub fn le(&self, other: &Rhs) -> bool
1.0.0[src]
#[must_use]pub fn gt(&self, other: &Rhs) -> bool
1.0.0[src]
#[must_use]pub fn ge(&self, other: &Rhs) -> bool
1.0.0[src]
impl Serialize for Mask
[src]
fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
__S: Serializer,
[src]
__S: Serializer,
impl StructuralPartialEq for Mask
[src]
Auto Trait Implementations
impl RefUnwindSafe for Mask
impl Send for Mask
impl Sync for Mask
impl Unpin for Mask
impl UnwindSafe for Mask
Blanket Implementations
impl<T> Any for T where
T: 'static + ?Sized,
[src]
T: 'static + ?Sized,
impl<T> Borrow<T> for T where
T: ?Sized,
[src]
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
[src]
T: ?Sized,
pub fn borrow_mut(&mut self) -> &mut T
[src]
impl<T> DeserializeOwned for T where
T: for<'de> Deserialize<'de>,
[src]
T: for<'de> Deserialize<'de>,
impl<T> From<T> for T
[src]
impl<T, U> Into<U> for T where
U: From<T>,
[src]
U: From<T>,
impl<T> Pointable for T
pub const ALIGN: usize
type Init = T
The type for initializers.
pub unsafe fn init(init: <T as Pointable>::Init) -> usize
pub unsafe fn deref<'a>(ptr: usize) -> &'a T
pub unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T
pub unsafe fn drop(ptr: usize)
impl<T> ToOwned for T where
T: Clone,
[src]
T: Clone,
type Owned = T
The resulting type after obtaining ownership.
pub fn to_owned(&self) -> T
[src]
pub fn clone_into(&self, target: &mut T)
[src]
impl<T, U> TryFrom<U> for T where
U: Into<T>,
[src]
U: Into<T>,
type Error = Infallible
The type returned in the event of a conversion error.
pub fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>
[src]
impl<T, U> TryInto<U> for T where
U: TryFrom<T>,
[src]
U: TryFrom<T>,