Expand description
Checkpoint tokenizer: a thin wrapper over the Hugging Face tokenizers crate that loads
tokenizer.json directly and resolves Laya’s special token ids. Mirrors the special-id
resolution in laya-ts/src/tokenizer.ts.
Structs§
- Laya
Tokenizer - A loaded checkpoint tokenizer.