Enum BaseTokenizer 

#[non_exhaustive]
pub enum BaseTokenizer {
    Simple,
    Whitespace,
    Raw,
    Ngram {
        min: u32,
        max: u32,
    },
    Custom(String),
}

Base tokenizer that produces the initial token stream.

Custom is a passthrough for backend-native tokenizers such as "lindera/*" or "jieba/*" (which require external dictionaries).

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

Simple

Split on whitespace and punctuation (recommended default).

Whitespace

Split on whitespace only.

Raw

No tokenization; the whole field is one token.

Ngram

Character N-gram tokenizer with inclusive [min, max] gram lengths.

Fields

min: u32

Minimum gram length (must be >= 1 and <= max).

max: u32

Maximum gram length.

Custom(String)

Backend-native tokenizer name, passed through verbatim (e.g. "jieba/default").

Trait Implementations

impl Clone for BaseTokenizer

fn clone(&self) -> BaseTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BaseTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for BaseTokenizer

fn default() -> BaseTokenizer

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for BaseTokenizer

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Eq for BaseTokenizer

impl PartialEq for BaseTokenizer

fn eq(&self, other: &BaseTokenizer) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for BaseTokenizer

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for BaseTokenizer