tiktoken-stream 0.1.0

Streaming token counter for partial LLM responses. Accumulates token count across chunks without holding the full text. Pluggable estimator function. Zero deps.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# tiktoken-stream

[![crates.io](https://img.shields.io/crates/v/tiktoken-stream.svg)](https://crates.io/crates/tiktoken-stream)
[![docs.rs](https://img.shields.io/docsrs/tiktoken-stream)](https://docs.rs/tiktoken-stream)

Streaming token counter for partial LLM responses. Pluggable estimator,
zero deps. Use it for progress bars, soft caps, and live cost displays
that update as deltas arrive.

## Usage

```rust
use tiktoken_stream::TokenStream;

let mut s = TokenStream::new(); // default: 4-chars-per-token ceil
for delta in ["Hello, ", "world!"] {
    let count = s.push(delta);
    println!("~{count} tokens so far");
}
```

Plug in tiktoken (or anything else) for accuracy:

```rust
use tiktoken_stream::TokenStream;

let mut tok = /* your tokenizer */;
let mut s = TokenStream::with_estimator(move |chunk: &str| {
    tok.encode_ordinary(chunk).len() as u64
});
```

## License

MIT or Apache-2.0.