pub fn run_pretok_program(program: &PreTokProgram, text: &str) -> Vec<String>Expand description
Execute program against text, returning the same piece sequence
the legacy regex pre-tokenizer would have emitted.
pub fn run_pretok_program(program: &PreTokProgram, text: &str) -> Vec<String>Execute program against text, returning the same piece sequence
the legacy regex pre-tokenizer would have emitted.