morpheme

Fast, pure-Rust subword tokenization, compatible with Hugging Face tokenizers.

morpheme loads, runs, trains and saves tokenizers in the Hugging Face tokenizer.json format — BPE, WordPiece, WordLevel and Unigram — and matches the reference implementation on the supported configurations and pinned real-model tests. Deliberate edge-case corrections and unsupported forms are documented in the compatibility guide.

• Compatible — reads and writes tokenizer.json; files trained with morpheme load in Python tokenizers and encode identically, and vice versa.
• Verified — every release is checked against 11 real, pinned tokenizers (BERT, GPT-2, RoBERTa, GPT-NeoX, Qwen2.5, Llama, T5, ALBERT, XLM-R, …) across tricky Unicode inputs.
• Pure Rust — no FFI, no C/C++ dependencies.
• Fast — faster than Python tokenizers in our encode and training benchmarks (see benchmarks).

Status

v0.3.0 — correctness fixes from a full code review (no panics on loadable files, complete truncation overflow, hardened Hub cache), legacy JSON forms, and faster encoding. The library API may still change before 1.0.

cargo add morpheme                    # the library
cargo add morpheme --features hub     # + download tokenizers from the Hugging Face Hub

The morpheme CLI:

cargo install morpheme-cli
# or a prebuilt binary (macOS, Linux, Windows):
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/sourceblender/morpheme/releases/latest/download/morpheme-cli-installer.sh | sh

On Windows: powershell -ExecutionPolicy Bypass -c "irm https://github.com/sourceblender/morpheme/releases/latest/download/morpheme-cli-installer.ps1 | iex".

Library

Load a supported tokenizer.json and encode:

use morpheme::Tokenizer;

let tokenizer = Tokenizer::from_file("bert-base-uncased/tokenizer.json")?;

let encoding = tokenizer.encode("Hello, world!", true)?;
assert_eq!(encoding.tokens(), ["[CLS]", "hello", ",", "world", "!", "[SEP]"]);
assert_eq!(encoding.ids(), [101, 7592, 1010, 2088, 999, 102]);
assert_eq!(encoding.offsets()[1], (0, 5)); // byte offsets into the input

let pair = tokenizer.encode(("How are you?", "Fine."), true)?;
println!("{:?}", pair.type_ids()); // [0, 0, 0, 0, 0, 0, 1, 1, 1]

let text = tokenizer.decode(encoding.ids(), true)?;
assert_eq!(text, "hello, world!");

With the hub feature, load straight from the Hugging Face Hub. Files go to the standard Hugging Face cache (shared with Python) and HF_TOKEN, HF_HOME, HF_ENDPOINT and HF_HUB_OFFLINE work as usual:

use morpheme::{FromPretrainedParameters, Tokenizer};

let tokenizer = Tokenizer::from_pretrained("google-bert/bert-base-uncased", None)?;
let pinned = Tokenizer::from_pretrained(
    "openai-community/gpt2",
    Some(FromPretrainedParameters::default().revision("607a30d783dfa663caf39e06633721c8d4cfcd7e")),
)?;

encode returns byte offsets; encode_char_offsets returns char offsets like the Python API. encode_batch encodes in parallel and applies the file's padding/truncation settings.

Train a GPT-2-style byte-level BPE tokenizer:

use morpheme::models::Bpe;
use morpheme::pre_tokenizers::ByteLevel;
use morpheme::trainers::BpeTrainer;
use morpheme::{AddedToken, Tokenizer};

let mut tokenizer = Tokenizer::new(Bpe::default())
    .with_pre_tokenizer(ByteLevel::new(false, true, true))
    .with_decoder(ByteLevel::default());

let trainer = BpeTrainer::builder()
    .vocab_size(5_000)
    .initial_alphabet(ByteLevel::alphabet())
    .special_tokens(vec![AddedToken::new("<|endoftext|>", true)])
    .build()?;
tokenizer.train_from_files(trainer, &["corpus.txt"])?;
tokenizer.save("tokenizer.json", true)?;

let ids = tokenizer.encode("Any text — even 😀 — round-trips.", false)?.ids().to_vec();
assert_eq!(tokenizer.decode(&ids, false)?, "Any text — even 😀 — round-trips.");

CLI

cargo install morpheme-cli   # or, from a checkout: cargo install --path apps/morpheme-cli

# Train (presets: byte-level for bpe, bert for wordpiece,
# sentencepiece for unigram, whitespace for wordlevel)
morpheme train --model bpe --vocab-size 5000 --out tokenizer.json examples/corpus.txt

morpheme encode -t tokenizer.json "the quick brown fox"
morpheme encode -t tokenizer.json --json --pair "second" "first"
morpheme decode -t tokenizer.json 84,259,420,1209,513   # → "the quick brown fox"
morpheme inspect -t tokenizer.json

# Budget raw text without configured padding or truncation
morpheme count -t tokenizer.json --no-special-tokens "the quick brown fox"
# Process bounded JSONL batches with one tokenizer load
morpheme encode-batch -t tokenizer.json --input documents.jsonl --ignore-tokenizer-settings
morpheme inspect -t tokenizer.json --json

# Any Hugging Face Hub model id works in place of a path (cached locally)
morpheme encode -t google-bert/bert-base-uncased "Hello, world!"
morpheme inspect -t meta-llama/Llama-3.2-1B --revision main   # gated: set HF_TOKEN

Every command uses the full pipeline stored in the file; --help on any command lists its options. See CLI automation for JSONL schemas, counting semantics, size limits, and streaming error behavior.

What's supported

Known differences from Hugging Face are listed in docs/interop.md.

How it's tested

• tests/hf_golden.rs — 11 real tokenizers pinned by revision (scripts/hf-fixtures.txt); ids, tokens, offsets, masks, word ids, decoding, pairs, batches and the serialized JSON must match outputs recorded from tokenizers 0.23.2 (scripts/gen_golden.py).
• scripts/check_python_interop.py — the reverse direction: morpheme-trained files loaded by Python.
• Property tests (lossless byte-level round-trips, valid offsets), per-component ground-truth tests, and regression tests for past bugs.

./scripts/fetch-hf-fixtures.sh   # once: downloads the pinned tokenizer.json files
cargo test --workspace

Project layout

morpheme/
├── crates/morpheme/     # the library
├── apps/morpheme-cli/   # the `morpheme` binary
├── docs/                # design and module documentation
├── examples/            # sample corpus and trained tokenizers
└── scripts/             # fixture download, golden generation, interop check

Documentation

• docs/architecture.md — design and pipeline.
• docs/modules/ — per-component reference.
• docs/interop.md — format compatibility and known differences.
• docs/benchmarks.md — performance and methodology.
• docs/document-workflow.md — executable dataset preparation with exact budgets and atomic publication.
• docs/roadmap.md — what's next.
• CONTRIBUTING.md, CHANGELOG.md.

Acknowledgements

morpheme's file format and component semantics follow Hugging Face tokenizers (Apache-2.0), which served as the behavioral reference.

License

MIT © 2026 The morpheme authors.