Module tokenizer 

Tokenizer for LFM2.5 models

Loads and uses the tokenizer.json from the model directory.

Structs

Sampler

Next-token sampler: temperature, top-k and nucleus (top-p) filtering.

Tokenizer

Wraps a Hugging Face tokenizer.json with special-token ids.

Functions

argmax

Index of the largest logit (NaN-safe).