Function ra_ap_syntax::tokenize [−][src]
pub fn tokenize(text: &str) -> (Vec<Token>, Vec<SyntaxError>)
Break a string up into its component tokens. Beware that it checks for shebang first and its length contributes to resulting tokens offsets.