Function nu_parser::lex

pub fn lex(
    input: &str, 
    span_offset: usize, 
    newline_mode: NewlineMode
) -> (Vec<Token>, Option<ParseError>)

Breaks the input string into a vector of tokens. This tokenization only tries to classify separators like semicolons, pipes, etc from external bare values (values that haven’t been classified further) Takes in a string and and offset, which is used to offset the spans created (for when this function is used to parse inner strings)