Function papyri_lang::parser::tokenize

source · 
pub fn tokenize(
    src: Rc<SourceFile>,
    strip_comments: bool,
    diagnostics: &mut Diagnostics
) -> Vec<Token>
Expand description

Tokenizes the given Papyri source file.