Function estimate_tokens 

pub fn estimate_tokens(text: &str) -> Result<usize>

Estimates the number of tokens in the given text using the OpenAI tokenizer. This uses the cl100k encoding which is used by GPT-3.5 and GPT-4.

Examples

use rusty_commit::utils::token::estimate_tokens;

let text = "Hello, world!";
let tokens = estimate_tokens(text).unwrap();
assert!(tokens > 0);