bitcannon 0.1.0-dev.0

SIMD Unicode atom classification + bitstream pre-tokenization: one classify pass into byte-exact token spans (cl100k, o200k, GPT-2/ByteLevel, DeepSeek, Whitespace, Bert, ...).
Documentation 
1
2
3
4
5
6
7
8
9

//! Per-arch kernels. Every one of these is PURE PERF and byte-exact with a portable path that is
//! always compiled, so correctness never depends on a kernel being present:
//!   - [`neon`] / [`x86`] build the block's bitstreams; `build_block_scalar` is the reference.
//!
//! (`classify` has its own kernels, next to the tables they index.)
#[cfg(target_arch = "aarch64")]
pub(crate) mod neon;
#[cfg(target_arch = "x86_64")]
pub(crate) mod x86;