Skip to main content

run_pretok_program

Function run_pretok_program 

Source
pub fn run_pretok_program(program: &PreTokProgram, text: &str) -> Vec<String>
Expand description

Execute program against text, returning the same piece sequence the legacy regex pre-tokenizer would have emitted.