Crate morpheme 

morpheme — fast, pure-Rust subword tokenization compatible with Hugging Face tokenizers.

morpheme loads, runs, trains and saves tokenizers in the Hugging Face tokenizer.json format (BPE, WordPiece, WordLevel and Unigram) and produces the same ids, tokens, offsets and decoded text as the reference implementation, apart from documented edge-case corrections.

Quick start

Load a tokenizer.json (from disk, or from the Hub with the hub feature) and encode:

use morpheme::Tokenizer;

let tokenizer = Tokenizer::from_file("tokenizer.json")?;
let encoding = tokenizer.encode("Hello, world!", true)?;
println!("{:?} {:?}", encoding.tokens(), encoding.ids());
let text = tokenizer.decode(encoding.ids(), true)?;

Or build one in code, train it, and save it:

use morpheme::models::WordPiece;
use morpheme::normalizers::BertNormalizer;
use morpheme::pre_tokenizers::BertPreTokenizer;
use morpheme::trainers::WordPieceTrainer;
use morpheme::{AddedToken, Tokenizer};

let mut tokenizer = Tokenizer::new(WordPiece::default())
    .with_normalizer(BertNormalizer::default())
    .with_pre_tokenizer(BertPreTokenizer);
let trainer = WordPieceTrainer::builder()
    .vocab_size(100)
    .special_tokens(vec![AddedToken::new("[UNK]", true)])
    .show_progress(false)
    .build()?;
tokenizer.train(trainer, ["Hello world!", "hello again, World"].into_iter())?;

let encoding = tokenizer.encode("HELLO, world", false)?;
assert_eq!(encoding.tokens(), ["hello", ",", "world"]);
assert_eq!(encoding.offsets()[1], (5, 6)); // byte offsets into the input

let json = tokenizer.to_json(false)?; // a valid Hugging Face tokenizer.json
let reloaded = Tokenizer::from_json(&json)?;
assert_eq!(reloaded.encode("HELLO, world", false)?, encoding);

The pipeline

input ─▶ added tokens ─▶ Normalizer ─▶ PreTokenizer ─▶ Model
      ─▶ truncation ─▶ PostProcessor ─▶ padding ─▶ Encoding

ids ─▶ (skip special tokens) ─▶ Decoder ─▶ text

Stage	Trait	Built-in components
Normalize text	Normalizer	normalizers
Split into words	PreTokenizer	pre_tokenizers
Words → tokens	Model	models
Add special tokens, merge pairs	PostProcessor	processors
Tokens → text	Decoder	decoders
Learn a model	Trainer	trainers

Each module has a wrapper enum (NormalizerWrapper, ModelWrapper, …) that a Tokenizer stores and that (de)serializes with the Hugging Face "type" tag. Offsets always point into the original input, even after normalization, thanks to NormalizedString’s alignment tracking: Tokenizer::encode returns byte offsets and Tokenizer::encode_char_offsets returns char offsets (what the Python library returns).

Cargo features

Feature	Default	Enables
progressbar	yes	Trainer progress bars on stderr (show_progress)
parallel	yes	rayon for encode_batch, decode_batch, batch padding and the trainers; without it they run sequentially
hub	no	Tokenizer::from_pretrained: download from the Hugging Face Hub into the cache shared with Python (native targets only)

The crate builds for wasm32-unknown-unknown with any feature set; there, hub is a no-op and Tokenizer::save is compiled out (use Tokenizer::to_json).

Compatibility

Behavior is verified against 11 real tokenizers (BERT, GPT-2, RoBERTa, Llama, T5, XLM-R, …) recorded from tokenizers 0.23. Known, deliberate differences are listed in the compatibility guide.

Re-exports

pub use added_vocabulary::AddedToken;

pub use added_vocabulary::AddedVocabulary;

pub use decoders::DecoderWrapper;

pub use encoding::Encoding;

pub use encoding::PaddingDirection;

pub use encoding::TruncationDirection;

pub use error::Error;

pub use error::Result;

pub use models::ModelWrapper;

pub use normalized_string::NormalizedString;

pub use normalized_string::OffsetRange;

pub use normalized_string::SplitDelimiterBehavior;

pub use normalizers::NormalizerWrapper;

pub use pre_tokenized_string::OffsetType;

pub use pre_tokenized_string::PreTokenizedString;

pub use pre_tokenized_string::Split;

pub use pre_tokenizers::PreTokenizerWrapper;

pub use processors::PostProcessorWrapper;

pub use tokenizer::FromPretrainedParameters;hub

pub use tokenizer::DecodeStream;

pub use tokenizer::EncodeInput;

pub use tokenizer::InputSequence;

pub use tokenizer::PaddingParams;

pub use tokenizer::PaddingStrategy;

pub use tokenizer::Tokenizer;

pub use tokenizer::TruncationParams;

pub use tokenizer::TruncationStrategy;

pub use traits::Decoder;

pub use traits::Model;

pub use traits::Normalizer;

pub use traits::PostProcessor;

pub use traits::PreTokenizer;

pub use traits::Trainer;

Modules

added_vocabulary

Added tokens: strings (usually special tokens such as [CLS] or <|endoftext|>) that are matched verbatim in the input before the model sees it, so they are never split.

decoders

Decoders: turn tokens back into text.

encoding

The Encoding produced by crate::Tokenizer::encode.

error

Crate-wide error type.

models

Models: BPE, WordPiece, WordLevel and Unigram.

normalized_string

NormalizedString: a string that remembers where every byte came from.

normalizers

Normalizers: rewrite the input text (lowercasing, Unicode normalization, accent stripping, …) while keeping alignments to the original so offsets stay exact.

pattern

Patterns used to find matches inside a string (for splitting and replacing), plus SysRegex, the regex engine used throughout the crate.

pre_tokenized_string

PreTokenizedString: the input split into pieces, each piece a NormalizedString that may already carry its tokens.

pre_tokenizers

Pre-tokenizers: split normalized text into the pieces (“words”) the model tokenizes independently.

processors

Post-processors: add special tokens and combine sequence pairs.

tokenizer

The Tokenizer: the full pipeline, plus loading and saving in the Hugging Face tokenizer.json format.

trainers

Trainers: learn a model from a text corpus.

traits

The five pipeline component traits.

Structs

Token

A token produced by a Model.

Constants

VERSION

Library version, mirrored from Cargo.toml.

Type Aliases

Offsets

A (start, end) byte or char range.