Crate piecer

Expand description

A tokenizer operating on Unicode codepoints. Supports automatic byte-fallback for out-of-vocabulary characters and with optional BPE merging mode.

The core vocabulary always includes the full byte range (0x00–0xff), special token ▁ plus any multi-byte Unicode codepoints found in the training data. During encoding, any input that cannot be mapped to multi-byte vocabulary elements decomposes into a sequence of single-byte tokens. On decoding, consecutive byte tokens that form valid UTF-8 are reassembled into characters; invalid sequences render as <hex> notation (e.g. <e9><be><8d>).

Unicode letters, digits, whitespace characters and “any symbols not belonging to these types” are guaranteed to never be mixed during a token merge. Words and standalone numbers are prepended by a special symbol ▁ internally to indicate a word or a number start. Numbers are split into parts consisting of no more than 3 digits. and optionally ▁, if it is in the beginning of the number.

§Quick start

use piecer::Tokenizer;

// Train a tokenizer with up to 512 merge operations
let tok: Tokenizer = Tokenizer::train("hello world", &["[PAD]"], Some(512));

let tokens = tok.encode("hello world");
let decoded = tok.decode(&tokens);
assert_eq!("hello world", decoded);

§Byte fallback

Characters absent from the training vocabulary are encoded as their UTF-8 byte tokens and transparently reassembled on decode:

use piecer::Tokenizer;

let tok: Tokenizer = Tokenizer::train("hello", &[], Some(512));
let tokens = tok.encode("龍");  // U+9F8D — not in training data
assert_eq!("龍", tok.decode(&tokens));  // reassembled from <e9><be><8d>

§Persistence

use piecer::Tokenizer;
use std::path::Path;

let tok: Tokenizer = Tokenizer::train("some training text", &[], Some(512));
tok.save(Path::new("my_tokenizer.json")).unwrap();
let loaded: Tokenizer = Tokenizer::load(Path::new("my_tokenizer.json")).unwrap();
assert_eq!(tok.vocab_size(), loaded.vocab_size());