Skip to main content

Module token

Module token 

Source

Functionsยง

estimate_tokens
Estimates the number of tokens in the given text using the OpenAI tokenizer. This uses the cl100k encoding which is used by GPT-3.5 and GPT-4.