Skip to main content

tokenize_with_comments

Function tokenize_with_comments 

Source
pub fn tokenize_with_comments(
    source: &str,
) -> Result<Vec<Spanned<Token>>, Vec<Spanned<LexerError>>>
Expand description

Tokenize source code, preserving comments.

Useful for pretty-printing or formatting tools that need to preserve comments.