Enum rust_tokenizers::Mask
source · pub enum Mask {
None,
Whitespace,
Punctuation,
CJK,
Special,
Begin,
Continuation,
Unfinished,
Unknown,
}
Expand description
Variants§
None
The token has no particular mask. This is the default situation. It may indicate that further processing can be done on a token.
Whitespace
The token represents a whitespace (in any shape or form)
Punctuation
The token represents punctuation (in any shape or form)
CJK
The token represents a single Chinese/Japanese/Korean character (including kana and hangul)
Special
The token is a special marker (such as a separator marker, a class marker, etc)
Begin
The token is the begin in a series of subtokens, the offset refers specifically to the sub-token. Subsequent tokens in this sequence will carry the ‘Continuation’ mask
Continuation
The token is the continuation of the previous token, the offset refers specifically to the sub-token. All but the first sub-token in a sequence carry this mask (the first carries ‘Begin’). (this is the reverse of Mask::Unfinished)
Unfinished
The token is the start of a token but not finished yet. All but the last sub-token in the a token sequence carry this mask. This is the reverse of Mask::Continuation.
Unknown
The token is out of vocabulary, it is unknown by the tokenizer and it will decode to unknown. Tokens that can be decoded properly (but may still be out of vocabulary) should not set this.
Trait Implementations§
source§impl<'de> Deserialize<'de> for Mask
impl<'de> Deserialize<'de> for Mask
source§fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where __D: Deserializer<'de>,
source§impl PartialEq for Mask
impl PartialEq for Mask
source§impl PartialOrd for Mask
impl PartialOrd for Mask
1.0.0 · source§fn le(&self, other: &Rhs) -> bool
fn le(&self, other: &Rhs) -> bool
self
and other
) and is used by the <=
operator. Read more