Expand description
Utilities for working with tokens and content
Functionsยง
- compare_
tokenization_ accuracy - Compare tiktoken accuracy against legacy estimation
- estimate_
tokens_ legacy - Estimate tokens using the legacy character-based method (for comparison)
- recommend_
token_ budget - Get recommended token budget based on model and content type