bitcannon 0.1.0-rc.3

SIMD Unicode atom classification + bitstream pre-tokenization: one classify pass into byte-exact token spans (cl100k, o200k, GPT-2/ByteLevel, DeepSeek, Whitespace, Bert, ...).
Documentation

Builds

bitcannon's sandbox limits

All the builds on docs.rs are executed inside a sandbox with limited resources. The limits for this crate are the following:

Available RAM 6.4 GB
Maximum rustdoc execution time 15m
Maximum size of a build log 102.4 kB
Network access blocked
Maximum number of build targets 10

If a build fails because it hit one of those limits please open an issue to get them increased.