simd-tokenizer 0.1.0

Safe SWAR-accelerated whitespace token estimator with an optional exact tiktoken (cl100k_base) backend.
Documentation