Function ra_ap_syntax::tokenize

pub fn tokenize(text: &str) -> (Vec<Token>, Vec<SyntaxError>)

Break a string up into its component tokens. Beware that it checks for shebang first and its length contributes to resulting tokens offsets.