pretokie 0.0.4

Fast, zero-allocation pretokenizers for BPE tokenizers

1
2
3

pub mod bert;
#[cfg(feature = "regex")]
pub mod regex;