Type Alias TokenizerFn 

pub type TokenizerFn = Box<dyn Fn(&PromptInput) -> Result<Vec<u32>, String> + Send + Sync>;

A tokenizer function that takes either chat messages or a raw prompt and returns tokenized output. This allows different implementations (tiktoken, transformers.js, etc.) to be passed in from the CLI or WASM interface. The tokenizer should apply the appropriate chat template for chat-style requests.

Aliased Type

pub struct TokenizerFn(/* private fields */);