llm-tokenizer 0.0.1

Tokenizer utilities for LLM inference

Coverage
100%
2 out of 2 items documented0 out of 1 items with examples
Size
Source code size: 1.52 kB This is the summed size of all the files inside the crates.io package for this release.
Documentation size: 1.04 MB This is the summed size of all files generated by rustdoc for all configured targets
Ø build duration
this release: 12s Average build duration of successful builds.
all releases: 1m 21s Average build duration of successful builds in releases after 2024-10-23.
Links
Repository
crates.io
Dependencies
Versions
Owners

llm-tokenizer

Tokenizer utilities for LLM inference

This crate is under development as part of the Shepherd project.

Full implementation coming soon.