Expand description
Fast pretokenizers for BPE tokenizers.
Each pretokenizer is a zero-allocation, single-pass iterator over text pieces.
§Example
use pretokie::Gpt2;
let pieces: Vec<&str> = Gpt2::new("Hello world").collect();
assert_eq!(pieces, vec!["Hello", " world"]);Modules§
- util
- Shared byte-level utilities.