token_processor 0.1.6

A fast, streaming‑first Rust library for processing LLM outputs by attaching callbacks to XML‑style tags—supporting both streaming and buffered handlers—and using aho‑corasick for ultra‑efficient, cross‑chunk pattern matching on decoded text tokens.
Documentation
Builds

token_processor's sandbox limits

All the builds on docs.rs are executed inside a sandbox with limited resources. The limits for this crate are the following:

Available RAM 6.44 GB
Maximum rustdoc execution time 15 minutes
Maximum size of a build log 102.4 kB
Network access blocked
Maximum number of build targets 10

If a build fails because it hit one of those limits please open an issue to get them increased.