Function nu_parser::lex

pub fn lex(input: &str, span_offset: usize) -> (Vec<Token>, Option<ParseError>)

Breaks the input string into a vector of tokens. This tokenization only tries to classify separators like semicolons, pipes, etc from external bare values (values that haven't been classified further) Takes in a string and and offset, which is used to offset the spans created (for when this function is used to parse inner strings)