morpheme 0.4.0

Fast, pure-Rust subword tokenization (BPE, WordPiece, Unigram) compatible with Hugging Face tokenizers.
Documentation 
1
2
3
4
5
6
7
8
9

//! Unigram language model (SentencePiece-style).

pub(crate) mod lattice;
mod model;
mod prng;
mod trie;

pub(crate) use lattice::Lattice;
pub use model::Unigram;