Module search_tokens 

Functions

count_tokens

Helper function to count tokens in a string using tiktoken (same tokenizer as GPT models)

get_tokenizer

Returns a reference to the tiktoken tokenizer