Struct TokenizerRegistry 

pub struct TokenizerRegistry { /* private fields */ }

Global tokenizer registry living on AppState.

Implementations

impl TokenizerRegistry

pub fn new( store: Arc<dyn TokenizerStore>, upstream: Arc<dyn TokenizerClient>, ) -> Self

pub fn set_download_enabled(&self, on: bool)

pub async fn list(&self) -> Vec<VocabInfo>

Builtins + bundled + persisted vocabs (admin surface; async because it asks the persistence backend).

pub fn resolve(&self, name: &str) -> Option<Arc<Tokenizer>>

memory → bundled name → None. Persisted/downloaded vocabs only show up after a background request_load hydrates them into memory; a miss here never blocks the request.

pub fn preheat(&self) -> LoadRequestStatus

Fire-and-forget warm-up of the bundled vocab on the blocking pool, so the first count request never pays the parse inline. Call once at boot.

pub fn request_load(&self, name: &str) -> LoadRequestStatus

Fire-and-forget load pipeline, deduped per name: hydrate from the persistence backend; when absent there, downloads are enabled, and the name is an HF repo path (org/repo), download hf.co/{name}/resolve/main/tokenizer.json through the shared upstream client and persist it. Never blocks the calling request.

pub async fn resolve_or_load( &self, name: &str, ) -> Result<Option<Arc<Tokenizer>>>

Resolve immediately or wait for persistence/download hydration.