Struct TokenOptimizer 

pub struct TokenOptimizer { /* private fields */ }

Implementations

impl TokenOptimizer

pub fn load_or_default(model_dir: &Path) -> Self

pub fn with_defaults() -> Self

pub fn optimize<'a>( &'a self, _concept: &str, representation: &'a str, ) -> &'a str

pub fn optimize_line(&self, line: &str) -> String

pub fn optimize_block(&self, content: &str) -> String

pub fn replacement_count(&self) -> usize

pub fn token_cost(text: &str) -> usize

BPE cost oracle: measure the actual token cost of a string representation. Used to pick the cheapest encoding when multiple are semantically equivalent.

pub fn cheaper_repr<'a>(a: &'a str, b: &'a str) -> &'a str

Choose the cheaper representation between two semantically equivalent strings.