eredu-text

eredu-text contains backend-neutral tokenizer and chat-template utilities for Rust language-model runtimes. It supports structured messages, system roles, tool metadata, Jinja chat templates, GGUF tokenizer metadata, Kimi-style tiktoken vocabularies, and the tokenizer backends used by eredu.

Most applications should depend on eredu, which exposes these capabilities through its higher-level loading and generation APIs. Use this crate directly when integrating the tokenizer or template layer into another runtime. It does not depend on a concrete backend or native accelerator runtime.

The portable pure-Rust fancy-regex tokenizer engine is always available, so --no-default-features is a valid configuration. Default features additionally enable Oniguruma (which the tokenizer selects when both engines are present) and the fast SentencePiece-compatible path.

License

Licensed under either Apache-2.0 or MIT.