tiktoken-stream

Streaming token counter for partial LLM responses. Pluggable estimator, zero deps. Use it for progress bars, soft caps, and live cost displays that update as deltas arrive.

Usage

use tiktoken_stream::TokenStream;

let mut s = TokenStream::new(); // default: 4-chars-per-token ceil
for delta in ["Hello, ", "world!"] {
    let count = s.push(delta);
    println!("~{count} tokens so far");
}

Plug in tiktoken (or anything else) for accuracy:

use tiktoken_stream::TokenStream;

let mut tok = /* your tokenizer */;
let mut s = TokenStream::with_estimator(move |chunk: &str| {
    tok.encode_ordinary(chunk).len() as u64
});

License

MIT or Apache-2.0.