bitcanon 0.1.0-dev.0

SIMD Unicode atom classification + bitstream pre-tokenization: one classify pass into byte-exact token spans (cl100k, o200k, GPT-2/ByteLevel, DeepSeek, Whitespace, Bert, ...).
Documentation
  • Coverage
  • 65.48%
    55 out of 84 items documented0 out of 26 items with examples
  • Size
  • Source code size: 309.0 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 1.8 MB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 5s Average build duration of successful builds.
  • all releases: 5s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • Homepage
  • huggingface/tokenizers
    11256 1268 76
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • ArthurZucker