pub type TokenizerFn = Box<dyn Fn(&[Message]) -> Result<Vec<u32>, String> + Send + Sync>;Expand description
A tokenizer function that takes messages and returns tokenized output. This allows different implementations (tiktoken, transformers.js, etc.) to be passed in from the CLI or WASM interface. The tokenizer should apply the appropriate chat template.
Aliased Typeยง
pub struct TokenizerFn(/* private fields */);