simd-tokenizer 0.1.2

Safe SWAR-accelerated whitespace token estimator with an optional exact tiktoken (cl100k_base, o200k_base) backend.
Documentation