Enum rust_tokenizers::Mask

pub enum Mask {
    None,
    Whitespace,
    Punctuation,
    CJK,
    Special,
    Begin,
    Continuation,
    Unfinished,
    Unknown,
}

Type indication for tokens (e.g. special token, white space, unknown…)

Variants

None

The token has no particular mask. This is the default situation. It may indicate that further processing can be done on a token.

Whitespace

The token represents a whitespace (in any shape or form)

Punctuation

The token represents punctuation (in any shape or form)

CJK

The token represents a single Chinese/Japanese/Korean character (including kana and hangul)

Special

The token is a special marker (such as a separator marker, a class marker, etc)

Begin

The token is the begin in a series of subtokens, the offset refers specifically to the sub-token. Subsequent tokens in this sequence will carry the ‘Continuation’ mask

Continuation

The token is the continuation of the previous token, the offset refers specifically to the sub-token. All but the first sub-token in a sequence carry this mask (the first carries ‘Begin’). (this is the reverse of Mask::Unfinished)

Unfinished

The token is the start of a token but not finished yet. All but the last sub-token in the a token sequence carry this mask. This is the reverse of Mask::Continuation.

Unknown

The token is out of vocabulary, it is unknown by the tokenizer and it will decode to unknown. Tokens that can be decoded properly (but may still be out of vocabulary) should not set this.

Trait Implementations

impl Clone for Mask

fn clone(&self) -> Mask

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Mask

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Mask

fn default() -> Mask

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for Mask

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl PartialEq<Mask> for Mask

fn eq(&self, other: &Mask) -> bool

This method tests for self and other values to be equal, and is used by ==.

#[must_use]

fn ne(&self, other: &Rhs) -> bool

This method tests for !=.

impl PartialOrd<Mask> for Mask

fn partial_cmp(&self, other: &Mask) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

#[must_use]

fn lt(&self, other: &Rhs) -> bool

This method tests less than (for self and other) and is used by the < operator.

#[must_use]

fn le(&self, other: &Rhs) -> bool

This method tests less than or equal to (for self and other) and is used by the <= operator.

#[must_use]

fn gt(&self, other: &Rhs) -> bool

This method tests greater than (for self and other) and is used by the > operator.

#[must_use]

fn ge(&self, other: &Rhs) -> bool

This method tests greater than or equal to (for self and other) and is used by the >= operator.

impl Serialize for Mask

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.

impl Copy for Mask

impl StructuralPartialEq for Mask