Claude tokenizer matching ai-tokenizer's lookup/BPE hybrid encoding.
- Coverage
- 100%
3 out of 3 items documented0 out of 0 items with examples - Size
- Source code size: 1.63 MB This is the summed size of all the files inside the crates.io package for this release.
- Documentation size: 314.55 kB This is the summed size of all files generated by rustdoc for all configured targets
- Ø build duration
- this release: 7s Average build duration of successful builds.
- all releases: 9s Average build duration of successful builds in releases after 2024-10-23.
- Links
- crates.io
- Dependencies
- Versions
-
- 0.42.0 (2026-06-26)
- 0.41.0 (2026-06-25)
- 0.40.3 (2026-06-25)
- 0.40.2 (2026-06-25)
- 0.40.1 (2026-06-24)
- 0.40.0 (2026-06-24)
- 0.39.4 (2026-06-18)
- 0.39.3 (2026-06-17)
- 0.39.2 (2026-06-15)
- 0.39.1 (2026-06-14)
- 0.39.0 (2026-06-13)
- 0.38.0 (2026-06-13)
- 0.37.2 (2026-06-12)
- 0.37.1 (2026-06-11)
- 0.37.0 (2026-06-10)
- 0.36.1 (2026-06-08)
- 0.36.0 (2026-06-07)
- 0.35.4 (2026-06-05)
- 0.35.3 (2026-06-04)
- 0.35.2 (2026-06-03)
- 0.35.1 (2026-06-03)
- 0.35.0 (2026-06-03)
- 0.34.0 (2026-06-01)
- 0.33.0 (2026-05-30)
- 0.32.0 (2026-05-27)
- 0.31.1 (2026-05-26)
- 0.31.0 (2026-05-26)
- 0.30.3 (2026-05-25)
- 0.30.2 (2026-05-25)
- 0.30.1 (2026-05-24)
- 0.30.0 (2026-05-24)
- 0.29.1 (2026-05-23)
- 0.29.0 (2026-05-22)
- 0.28.2 (2026-05-22)
- 0.28.1 (2026-05-22)
- 0.28.0 (2026-05-21)
- 0.27.1 (2026-05-21)
- 0.27.0 (2026-05-20)
- Owners
-