simd-tokenizer 0.1.3

Safe SWAR-accelerated whitespace token estimator with optional exact tiktoken backend
Documentation