Available on crate feature
tiktoken only.Expand description
Use tiktoken-rs to estimate token count on various common Swiftide types
Intended to be used for openai models.
Note that the library is heavy on the unwraps.
Structsยง
- TikToken
- A tiktoken based tokenizer for openai models. Can also be used for other models.