Skip to main content

Module tokenizer

ck_embed

Module tokenizer 

Source

Structs§

TokenEstimator
Simple token estimation for code and text This is a rough approximation since we don’t have access to the actual model tokenizer