sentencepiece_rust/lib.rs
1//! A from-scratch, pure-Rust port of [SentencePiece] **inference**
2//! (encode/decode), differentially verified against the upstream C++/Python
3//! implementation — the same harness used across the author's other library
4//! ports.
5//!
6//! This is **not** a wrapper around the C++ library (that already exists as the
7//! `sentencepiece` crate). The algorithms here are reimplemented in Rust from
8//! the reference source in `projects/Reference/sentencepiece`.
9//!
10//! # Status
11//! Inference is feature-complete for BPE and Unigram models, verified against
12//! the Python `sentencepiece` oracle:
13//! - **v0.1:** BPE segmentation + ASCII/whitespace normalisation.
14//! - **v0.2:** Darts charsmap normaliser → full Unicode input (full-width
15//! folding, ligatures, CJK, …).
16//! - **v0.3 (current):** Unigram Viterbi segmentation (the default model type),
17//! plus faithful decode (byte-piece reassembly, unk surface, dummy-prefix).
18//!
19//! Encode *and* decode are verified against the oracle across BPE, Unigram, and
20//! byte-fallback models (117 cases). Training is intentionally out of scope —
21//! this crate loads a trained `.model` and tokenises with it.
22//!
23//! # Example
24//! ```no_run
25//! use sentencepiece_rust::SentencePieceProcessor;
26//! let sp = SentencePieceProcessor::open("tests/models/botchan_1000_bpe.model")?;
27//! let ids = sp.encode("hello world")?;
28//! let text = sp.decode(&ids)?;
29//! # Ok::<(), sentencepiece_rust::Error>(())
30//! ```
31//!
32//! [SentencePiece]: https://github.com/google/sentencepiece
33
34mod bpe;
35mod error;
36mod model;
37mod normalizer;
38mod processor;
39mod proto;
40mod trie;
41mod unigram;
42mod vocab;
43
44pub use error::{Error, Result};
45pub use model::{ModelType, PieceType};
46pub use processor::SentencePieceProcessor;
47pub use vocab::Vocab;