toklen 0.1.0

A single-threaded, lightweight, and fast token counter.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101

# toklen

A single-threaded, lightweight, and efficient token counter for BPE tokenizers.

**toklen** is a focused Rust library that counts tokens in a text—without allocating
a token list, without decoding, and without generating token IDs. So it's faster.

It loads the standard **tokenizer.json** format used by HuggingFace
**[tokenizers](https://github.com/huggingface/tokenizers)**.
Same config, zero conversion—drop it in and start counting.

It's single-threaded by choice — no Async, no Rayon, no thread pools. Despite this, it is thread-safe:
all internal state is thread-local, so you can safely share the tokenizer across threads.

## Usage

Add to your `Cargo.toml`:

```toml
[dependencies]
toklen = "0.1"
```

Rust code:

```rust
use toklen::Tokenizer;

// Load a standard tokenizer.json (e.g. from a HuggingFace model).
let json = std::fs::read_to_string("tokenizer.json").unwrap();
let tokenizer = Tokenizer::from_json(&json).unwrap();

// Count tokens.
let count = tokenizer.encode_len("Hello, world!").unwrap();
println!("Token count: {count}");

// On failure, encode_len returns Err(estimate) —— input.len() / 4
// as a rough fallback that is always safe for progress reporting.
```

For CLI tools or WASM builds, embed the JSON at compile time:

```rust
const TOKENIZER_JSON: &[u8] = include_bytes!("tokenizer.json");

fn main() {
    let tokenizer = toklen::Tokenizer::from_json(TOKENIZER_JSON).unwrap();
    let count = tokenizer.encode_len("Some text to count").unwrap();
    println!("{count} tokens");
}
```

## API

| Method | Description |
|---|---|
| `Tokenizer::from_json(json)` | Build a tokenizer from `tokenizer.json` bytes. |
| `tokenizer.encode_len(text)` | Count tokens in `text`. Returns `Ok(count)` or `Err(estimate)`. |

The `encode_len` method never panics on valid UTF-8 input. Errors bubble up
from underlying regex or normalization steps; the fallback estimate
(`text.len() / 4`) lets callers degrade gracefully.

## Performance

Its single-threaded throughput is on par with multi-threaded [fastokens](https://docs.rs/crate/fastokens/latest/features)
and roughly 5× faster than [tokenizers](https://docs.rs/tokenizers/latest/tokenizers) on equivalent hardware.

See [`/bench`](bench/) for the benchmark suite and raw numbers.

**Note:**

- `toklen` is thread-safe and designed to be stored as a global static resource (e.g., with `LazyLock`), 
  allowing the tokenizer configuration to be reused across calls without re-initialization overhead.
- The first call to `encode_len()` incurs cold-start overhead (lazy initialization of internal caches).
  Subsequent calls are stable and fast—this behavior is consistent with other tokenizer implementations.

## How it works

```text
input text
  → added-token split (Aho–Corasick)
  → normalization (NFC, sequence)
  → pre-tokenization (ByteLevel, Split, regex)
  → BPE merge counting (priority-queue merge loop)
  → token count
```

## Supported `tokenizer.json` subset

| Section | Status |
|---|---|
| `added_tokens` | Full support (Aho–Corasick matching with memchr prefilter) |
| `normalizer` | NFC + Sequence. Other normalizers return an error. |
| `pre_tokenizer` | ByteLevel, Split (PCRE2 JIT + fancy_regex fallback), Sequence |
| `model` | **BPE only** — other model types error immediately |
| `decoder` | Not used (this is a counter, not a full tokenizer) |

## License

Licensed under either of [Apache License 2.0](LICENSE-APACHE) or [MIT License](LICENSE-MIT), at your option.