Enum Tokenizer 

pub enum Tokenizer {
    Word,
    Whitespace,
    Prefix,
    Multilingual,
}

Tokenizer, mirroring Qdrant’s TokenizerType names.

Multilingual parses but fails closed at build time: it needs the charabia/vaporetto segmentation stack, which qql-embed deliberately does not depend on (lean core for WASM/edge; no model files to ship).

Variants

Word

Split on non-alphanumeric boundaries (Qdrant default).

Whitespace

Split on Unicode whitespace only.

Prefix

Document side expands all n-grams; query side keeps the longest only.

Multilingual

Script-aware segmentation (requires tokenizer support not compiled in).

Implementations

impl Tokenizer

pub fn parse(name: &str) -> Result<Self, QqlError>

Parse a Qdrant tokenizer name ("word", "whitespace", "prefix", "multilingual"), ASCII-case-insensitively (superset of Qdrant’s case-sensitive spelling, same accepted set). Anything else fails closed.

pub fn name(self) -> &'static str

Canonical Qdrant spelling.

Trait Implementations

impl Clone for Tokenizer

fn clone(&self) -> Tokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Tokenizer

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Tokenizer

fn default() -> Tokenizer

Returns the “default value” for a type.

impl Eq for Tokenizer

impl PartialEq for Tokenizer

fn eq(&self, other: &Tokenizer) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Tokenizer