toktrie_tiktoken 1.8.0

tiktoken (OpenAI BPE) library support for toktrie and llguidance

Coverage
100%
7 out of 7 items documented0 out of 5 items with examples
Size
Source code size: 13.45 kB This is the summed size of all the files inside the crates.io package for this release.
Documentation size: 208.8 kB This is the summed size of all files generated by rustdoc for all configured targets
Ø build duration
this release: 10s Average build duration of successful builds.
all releases: 46s Average build duration of successful builds in releases after 2024-10-23.
Links
guidance-ai/llguidance
850 80 35
crates.io
Dependencies
Versions
Owners

This crate integrates the tiktoken BPE tokenizer (used by OpenAI models) with [toktrie], providing a [TokenizerEnv] implementation backed by tiktoken's [CoreBPE].