Structs§
- Tokenizer
- A tokenizer that can count tokens in text.
Enums§
- Tokenizer
Kind - Which tokenizer to use for token counting.
Functions§
- estimate_
tokens - Estimate the number of tokens for a piece of content using heuristic.
- is_
prose_ extension - Check if a file extension indicates prose content