Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(segment: &str) -> Vec<String>
Expand description

Naive shell tokenizer for arity lookup: split on whitespace, keeping quoted runs together (the quotes are stripped, matching how the arity table keys are written).