compare_tokenization_accuracy

scribe_core::tokenization::utils

Function compare_tokenization_accuracy 

Source 
pub fn compare_tokenization_accuracy(
    content: &str,
    counter: &TokenCounter,
) -> Result<TokenizationComparison>
Expand description

Compare tiktoken accuracy against legacy estimation