Struct TokenizerRegistry 

pub struct TokenizerRegistry { /* private fields */ }

Registry for named tokenizers

Allows registering tokenizers by name and retrieving them for use during indexing. Pre-registers common tokenizers: “default”, “simple”, “lowercase”, “en_stem”, etc.

Implementations

impl TokenizerRegistry

pub fn new() -> Self

Create a new tokenizer registry with default tokenizers registered

pub fn register<T: Tokenizer>(&self, name: &str, tokenizer: T)

Register a tokenizer with a name

pub fn get(&self, name: &str) -> Option<BoxedTokenizer>

Get a tokenizer by name

pub fn contains(&self, name: &str) -> bool

Check if a tokenizer is registered

pub fn names(&self) -> Vec<String>

List all registered tokenizer names

Trait Implementations

impl Clone for TokenizerRegistry

fn clone(&self) -> TokenizerRegistry

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for TokenizerRegistry

fn default() -> Self

Returns the “default value” for a type.