Skip to main content

Crate dynamo_tokenizers

Crate dynamo_tokenizers 

Source

Re-exports§

pub use basetenkenizer::BasetenTokenizer;
pub use cache::CacheTokenUsage;
pub use cache::CacheTokenUsageFn;
pub use cache::CachedTokenizer;
pub use cache::L1CacheStats;
pub use cache::SharedTokenizerCache;
pub use cache::SharedTokenizerCacheStats;
pub use fastokens::FastTikTokenTokenizer;
pub use fastokens::FastTokenizer;
pub use hf::HuggingFaceTokenizer;
pub use tekken::TekkenTokenizer;
pub use tiktoken::TikTokenTokenizer;
pub use traits::DecodeResult;

Modules§

basetenkenizer
Baseten Tokenizer backend for high-performance BPE encoding and decoding.
cache
Tokenizer caching layer (L1: prefix matching at special-token boundaries).
fastokens
Fastokens backend using the fastokens crate for high-performance BPE encoding.
hf
tekken
Mistral Tekken tokenizer (tekken.json).
tiktoken
traits

Structs§

DecodeStream
DecodeStream will keep the state necessary to produce individual chunks of strings given an input stream of token_ids.
EncodeSegment
A rendered prompt segment with an explicit trust boundary for special tokens.
Error
The Error type, a wrapper around a dynamic error type.
Sequence
Maintains state for an ongoing sequence of tokens and their decoded text
StopSequenceDecoder
A Sequence for decoding a stream of token ids into text and detecting stop sequences. A stop sequence is either a matching token_id or a sequence of texts/strings which match. Matches happen first at the token-level, then at the sequence-level. Hidden takes precedence over visible. For example, if you put the same token_id in both stop_token_ids_visible and stop_token_ids_hidden, the token_id will be treated as hidden.
StopSequenceDecoderBuilder
Tokenizer
Main tokenizer wrapper that provides a unified interface for different tokenizer implementations
TokenizerOptions
Construction options for Tokenizer::from_file_with_options / create_tokenizer_from_file, applied to concrete tokenizers via traits::Tokenizer::with_options.

Enums§

Encoding
Contains the results of tokenizing text: token IDs, string tokens, and their spans
SequenceDecoderOutput
The output conditions/values of a SequenceDecoder::add_token_id operation. Result of decoding a token, indicating whether text was produced or a stop condition was met
TokenizerType
Represents the type of tokenizer being used

Functions§

create_tokenizer_from_file
Create a tokenizer from a file path to a tokenizer file. The file extension is used to determine the tokenizer type. Supported file types are:
create_tokenizer_from_file_with_options
Create a tokenizer from a file path to a tokenizer file with additional tokenizer option. The file extension is used to determine the tokenizer type. Supported file types are:
file_json_field
log_json_err

Type Aliases§

Offsets
character offsets in the original text
Result
Result<T, Error>
TokenIdType