splintr 0.19.1

Fast Rust tokenizer (BPE + SentencePiece + WordPiece) with Python bindings
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

//! HuggingFace pre-tokenizer pipeline.
//!
//! HF pre-tokenizers form an ordered pipeline: each stage takes the current list
//! of string pieces and splits each one further. A single regex can't model that
//! (e.g. Falcon = `Punctuation → ByteLevel → Digits → Split`), so this engine
//! applies the stages in order and returns the final pre-token pieces.
//!
//! When a `ByteLevel` stage is present the pieces come out byte-level-encoded
//! (each byte mapped to a printable code point), ready for BPE against a
//! byte-level vocab — so the consumer must NOT byte-level-encode again.
//!
//! The pipeline is split by role: `spec` is the public description a caller
//! builds, `stage` is its compiled counterpart, `split` holds the splitters
//! every stage is made of, `pipeline` runs them in order, and `parse` reads
//! a HuggingFace `pre_tokenizer` JSON node into a spec. Those modules are
//! private, so they are named here rather than linked.

mod parse;
mod pipeline;
mod spec;
mod split;
mod stage;
#[cfg(test)]
mod tests;

pub(crate) use parse::parse;
pub use pipeline::PreTokenizer;
pub use spec::{PreTokStage, SplitBehavior, SplitPattern};
pub(crate) use split::estimated_pieces;