pub fn lex(source: &str, media_type: MediaType) -> Vec<LexedItem>
Given the source text and media type, tokenizes the provided text to a collection of tokens and comments.