tpt-tokenizer-core 0.1.0

Pure-Rust BPE and WordPiece tokenizer — no_std + alloc compatible, zero dependencies.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

# tpt-tokenizer-core

[![crates.io](https://img.shields.io/crates/v/tpt-tokenizer-core.svg)](https://crates.io/crates/tpt-tokenizer-core)
[![docs.rs](https://img.shields.io/docsrs/tpt-tokenizer-core)](https://docs.rs/tpt-tokenizer-core)
[![License: MIT OR Apache-2.0](https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-blue.svg)](#license)

A small, **dependency-free** implementation of the two tokenization schemes used
by most open-weight LLMs, written in pure Rust:

- [`BpeTokenizer`] — Byte-Pair Encoding (GPT-2 / Llama style).
- [`WordPieceTokenizer`] — WordPiece (BERT style).

Both implement the shared `Tokenizer` trait (`encode` / `decode`).

## Features

- **`tokenizer.json` loading** — parse the modern unified Hugging Face format
  (`from_tokenizer_json_str` / `from_tokenizer_json_file`) with a built-in JSON
  parser, no `serde`. Auto-detects byte-level BPE, lowercasing, and special
  `added_tokens`.
- **Batch encoding** — BOS/EOS insertion, padding, and truncation via
  `EncodeConfig` + `TokenizerExt::encode_with` / `encode_batch`, returning an
  `Encoding { ids, attention_mask }`.
- **Byte-level BPE** — `BpeTokenizer::with_byte_level()` guarantees encoding
  never fails on arbitrary UTF-8 and `decode(encode(s)) == s`.
- **Special tokens** — register atomic markers (`<|endoftext|>`, `<s>`) that are
  never split.
- **Unicode normalization** (opt-in) — NFC/NFD/NFKC/NFKD via the `normalization`
  feature (see below).

## `no_std` + `alloc`

The tokenization logic is `#![no_std]` compatible: it only depends on `alloc`
and never touches the standard library. The `std` feature (enabled by default)
adds convenience constructors (`BpeTokenizer::from_files`,
`WordPieceTokenizer::from_file`, `from_tokenizer_json_file`) that load from disk.

```toml
# Fully `no_std` (you supply the vocab/merges at runtime):
tpt-tokenizer-core = { version = "0.1.0", default-features = false }
```

### Optional Unicode normalization

Correct NFC/NFKC needs the Unicode Character Database, so it is gated behind the
opt-in `normalization` feature (which pulls in `unicode-normalization`) to keep
the default build dependency-free:

```toml
tpt-tokenizer-core = { version = "0.1.0", features = ["normalization"] }
```

```rust
use tpt_tokenizer_core::{BpeTokenizer, NormalizationForm};

let tok = BpeTokenizer::from_vocab_merges(vocab, merges)
    .with_normalization(NormalizationForm::Nfc);
```

## Usage

```rust
use std::collections::BTreeMap;
use tpt_tokenizer_core::{BpeTokenizer, Tokenizer};

let mut vocab = BTreeMap::new();
vocab.insert("l".to_string(), 0u32);
vocab.insert("o".to_string(), 1u32);
vocab.insert("w".to_string(), 2u32);
vocab.insert("lo".to_string(), 3u32);
vocab.insert("low".to_string(), 4u32);
let merges = vec![
    ("l".to_string(), "o".to_string()),
    ("lo".to_string(), "w".to_string()),
];
let tok = BpeTokenizer::from_vocab_merges(vocab, merges);
assert_eq!(tok.encode("low").unwrap(), vec![4]);
```

## License

Licensed under either of MIT or Apache-2.0 at your option.