pub fn cl100k_base_singleton() -> &'static CoreBPEExpand description
Returns a singleton instance of the cl100k_base tokenizer.
Use for ChatGPT models, text-embedding-ada-002
This function will only initialize the tokenizer once, and then return a reference the tokenizer