Crate tpt_tokenizer_core 

tpt-tokenizer-core

A small, dependency-free implementation of the two tokenization schemes used by most open-weight LLMs:

• BpeTokenizer — Byte-Pair Encoding (GPT-2 / Llama style).
• WordPieceTokenizer — WordPiece (BERT style).

Both implement the shared Tokenizer trait with encode / decode.

BpeTokenizer additionally supports GPT-2 byte-level mode (with_byte_level) — encoding never fails on arbitrary UTF-8 and decode(encode(s)) == s — and atomic special tokens (with_special_tokens). WordPieceTokenizer runs a BERT-style basic tokenizer (whitespace + punctuation splitting, CJK isolation, optional with_lowercase).

The crate is #![no_std] compatible: all tokenization logic lives behind alloc and never touches the standard library. The std feature (enabled by default) only adds convenience constructors that load vocabularies and merge tables from disk.

Example

use tpt_tokenizer_core::{BpeTokenizer, Tokenizer};

let mut vocab = std::collections::BTreeMap::new();
vocab.insert("l".to_string(), 0u32);
vocab.insert("o".to_string(), 1u32);
vocab.insert("w".to_string(), 2u32);
vocab.insert("lo".to_string(), 3u32);
vocab.insert("low".to_string(), 4u32);
let merges = vec![
    ("l".to_string(), "o".to_string()),
    ("lo".to_string(), "w".to_string()),
];
let tok = BpeTokenizer::from_vocab_merges(vocab, merges);
let ids = tok.encode("low").unwrap();
assert_eq!(ids, vec![4]);

Re-exports

pub use encoding::EncodeConfig;

pub use encoding::Encoding;

pub use encoding::Padding;

pub use encoding::TokenizerExt;

pub use encoding::Truncation;

pub use error::TokenizerError;

pub use loader::from_tokenizer_json_file;

pub use loader::from_tokenizer_json_str;

pub use loader::LoadedTokenizer;

pub use tokenizer::TokenId;

pub use tokenizer::Tokenizer;

Modules

encoding

Batch encoding, special-token insertion, padding, and truncation.

error

Error type for tokenization and tokenizer loading.

json

A minimal, dependency-free JSON parser.

loader

Loader for the modern unified Hugging Face tokenizer.json format.

tokenizer

Shared tokenizer trait and token types.

Structs

BpeTokenizer

A Byte-Pair Encoding tokenizer built from a vocabulary and an ordered list of merge rules.

WordPieceTokenizer

A WordPiece tokenizer built from a vocabulary.