toklen 0.2.0

A single-threaded, lightweight, and efficient token counter.

**toklen** counts tokens in a text — no token-list allocation,
no token IDs, no decoding — so it's fast and small.

It's single-threaded by choice — no async, no Rayon. Despite this, it is thread-safe:
all internal state is thread-local, so you can safely share the tokenizer across threads.

It loads the standard **tokenizer.json** format used by HuggingFace
**[tokenizers](https://github.com/huggingface/tokenizers)**.
Same config, zero conversion — drop it in and start counting.

## Usage

Add to your `Cargo.toml`:

```toml
[dependencies]
toklen = "0.2"
```

Rust code:

```rust,no_run
use toklen::Tokenizer;

// Load a standard tokenizer.json (e.g. from a HuggingFace model).
let json = std::fs::read_to_string("tokenizer.json").unwrap();
let tokenizer = Tokenizer::from_json(&json).unwrap();

// Count tokens.
let count = tokenizer.encode_len("Hello, world!").unwrap();
println!("Token count: {count}");

// On failure, encode_len returns Err(estimate) — input.len() / 4,
// a rough fallback that is always safe for progress reporting.
```

For CLI tools or WASM builds, embed the JSON at compile time:

```rust,ignore
const TOKENIZER_JSON: &[u8] = include_bytes!("tokenizer.json");

fn main() {
    let tokenizer = toklen::Tokenizer::from_json(TOKENIZER_JSON).unwrap();
    let count = tokenizer.encode_len("Some text to count").unwrap();
    println!("{count} tokens");
}
```

## API

| Method | Description |
|---|---|
| `Tokenizer::from_json(json)` | Build a tokenizer from `tokenizer.json` bytes. |
| `tokenizer.encode_len(text)` | Count tokens in `text`. Returns `Ok(count)` or `Err(estimate)`. |

`encode_len` never panics. Errors from the underlying regex or normalization
steps are converted into `Err(estimate)` (`text.len() / 4`), so callers can
always degrade gracefully.

## Performance

toklen's single-threaded throughput is on par with multi-threaded [fastokens](https://docs.rs/fastokens)
and roughly 5× faster than [tokenizers](https://docs.rs/tokenizers) on equivalent hardware.

See [`/bench`](bench/) for the benchmark suite and raw numbers.

**Note:**

`toklen` is thread-safe and designed to be stored as a global resource (e.g. `static LazyLock<_>`), 
allowing the tokenizer configuration to be reused across calls without re-initialization overhead.

## How it works

```text
input text
  → added-token split (Aho–Corasick)
  → normalization (Sequence: NFC/NFD/…, Lowercase, StripAccents, Replace, …)
  → pre-tokenization (ByteLevel, Split regex, Whitespace, …)
  → model counting (BPE priority-queue merge loop, or WordLevel lookup)
  → token count
```

## Supported **tokenizer.json** subset

`toklen` reads the `tokenizer.json` format produced by HuggingFace
[tokenizers](https://huggingface.co/docs/tokenizers/python/latest/index.html).

The components and their semantics are documented in the
[tokenizers component reference](https://huggingface.co/docs/tokenizers/python/latest/components.html).
Configs that use an unsupported (or disabled) type return an error from
`Tokenizer::from_json`.

### added_tokens

Fully supported — added tokens are matched with Aho–Corasick (plus a memchr
prefilter), and each match counts as exactly one token.
Always compiled in (no feature gate).

### normalizer

| Type | Cargo feature |
|---|---|
| Sequence | always available |
| NFC | `nfc` |
| NFD | `nfd` |
| NFKC | `nfkc` |
| NFKD | `nfkd` |
| Lowercase | `lowercase` |
| StripAccents | `strip_accents` (pulls in `icu_properties`) |
| Strip | `strip` |
| Replace | `replace` |
| Prepend | `prepend` |
| ByteLevel | `byte_level` |

**Note:** *BertNormalizer* is not supported yet.

### pre_tokenizer

| Type | Cargo feature |
|---|---|
| Sequence | always available |
| ByteLevel | `byte_level` |
| Split | `split` (PCRE2 JIT + fancy_regex fallback) |
| Whitespace | `whitespace` |
| WhitespaceSplit | `whitespace_split` |
| Punctuation | `punctuation` |
| Metaspace | `metaspace` |
| Digits | `digits` |
| CharDelimiterSplit | `char_delimiter_split` |

### model

| Type | Cargo feature |
|---|---|
| BPE | `bpe` |
| WordLevel | `word_level` |

**Note:** *Unigram* and *WordPiece* are not supported yet.

### decoder / post_processor / other

Ignored — `toklen` only counts the tokens produced by the model, so decoding
and post-processing never run. Counts therefore correspond to
`encode(text, add_special_tokens=False)`: special tokens inserted by a
post-processor (e.g. `[CLS]`/`[SEP]`) are not included.

### Cargo features

Everything above is enabled by default (`default = ["full"]`). Each component
sits behind a cargo feature of the same name, so you can turn off whatever your
tokenizer configs never use — fewer compiled variants shrink the library and
tighten the dispatch `match` on the hot path:

```toml
[dependencies]
# For example, only what a GPT-style BPE tokenizer needs:
toklen = { version = "0.2", default-features = false, features = ["bpe", "byte_level", "split"] }
```

The meta features `normalizers`, `pre_tokenizers`, and `models` enable a whole
group at once; `full` (the default) enables all three.

## License

Licensed under either of [Apache License 2.0](LICENSE-APACHE) or [MIT License](LICENSE-MIT), at your option.