Module utils

Module utils 

Source
Expand description

Utilities for working with tokens and content

Functionsยง

compare_tokenization_accuracy
Compare tiktoken accuracy against legacy estimation
estimate_tokens_legacy
Estimate tokens using the legacy character-based method (for comparison)
recommend_token_budget
Get recommended token budget based on model and content type