Struct TokenizerRegistry 

pub struct TokenizerRegistry { /* private fields */ }

Registry for managing tokenizers keyed by UUID

Features:

• Thread-safe concurrent access using DashMap
• Per-key locking to prevent duplicate loading
• Lookup by UUID (primary) or name (secondary index)

Implementations

impl TokenizerRegistry

pub fn new() -> Self

Create a new empty registry

pub fn generate_id() -> String

Generate a new UUID for a tokenizer

pub async fn load<F, Fut>( &self, id: &str, name: &str, source: &str, loader: F, ) -> Result<LoadOutcome, LoadError>

where F: FnOnce() -> Fut, Fut: Future<Output = Result<Arc<dyn Tokenizer>, String>>,

Load and register a tokenizer

Validates inputs, handles deduplication, and loads the tokenizer if needed. Per-key locking ensures only one load happens per name, preventing race conditions.

Arguments

• id - Pre-generated UUID (use generate_id() to create one)
• name - User-provided name (used for deduplication, must not be empty)
• source - Source path or HuggingFace model ID (must not be empty)
• loader - Async function that loads the tokenizer

Returns

• Ok(LoadOutcome::Loaded { id }) - Tokenizer was newly loaded
• Ok(LoadOutcome::AlreadyExists { id }) - Tokenizer already existed
• Err(LoadError) - Validation failed or loading failed

pub fn get_by_id(&self, id: &str) -> Option<TokenizerEntry>

Get a tokenizer by UUID

pub fn get_by_name(&self, name: &str) -> Option<TokenizerEntry>

Get a tokenizer by name

pub fn get(&self, name_or_id: &str) -> Option<Arc<dyn Tokenizer>>

Get a tokenizer (for backward compatibility, tries name first then ID)

pub fn contains(&self, name: &str) -> bool

Check if a tokenizer is registered by name

pub fn contains_id(&self, id: &str) -> bool

Check if a tokenizer is registered by ID

pub fn len(&self) -> usize

Get the number of loaded tokenizers

pub fn is_empty(&self) -> bool

Check if the registry is empty

pub fn list(&self) -> Vec<TokenizerEntry>

List all registered tokenizers

pub fn remove_by_id(&self, id: &str) -> Option<TokenizerEntry>

Remove a tokenizer by ID

Returns the entry if it was present.

pub fn remove(&self, name: &str) -> Option<TokenizerEntry>

Remove a tokenizer by name

Returns the entry if it was present.

pub fn clear(&self)

Clear all tokenizers from the registry

Trait Implementations

impl Default for TokenizerRegistry

fn default() -> Self

Returns the “default value” for a type.