Struct CharApproxTokenizer 

pub struct CharApproxTokenizer;

Trivial tokenizer using the well-known “≈ 4 chars per token” rule of thumb. Default for the builder so the crate has no runtime model assets.

Counts bytes (not unicode scalar values) divided by 4, rounded up, with an extra +1 floor so any non-empty input reports at least 1 token. This matches what most sources will report in estimated_tokens, so the builder won’t surprise sources during pruning at default settings.

Not accurate for billing or budgeting against a real LLM provider — enable the tiktoken feature and use TiktokenCl100k when accuracy matters.

Implementations

impl CharApproxTokenizer

pub fn new() -> Self

Construct a new CharApproxTokenizer.

Trait Implementations

impl Clone for CharApproxTokenizer

fn clone(&self) -> CharApproxTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for CharApproxTokenizer

impl Debug for CharApproxTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CharApproxTokenizer

fn default() -> CharApproxTokenizer

Returns the “default value” for a type.

impl Tokenizer for CharApproxTokenizer

fn count(&self, text: &str) -> usize

Number of tokens the model would see for text.

fn encode(&self, _text: &str) -> Option<Vec<u32>>

Token IDs for providers that need them. Default returns None; only tokenizers that have a real BPE backing (e.g. tiktoken) override.