pub fn tokenize(source: &str) -> Vec<String>Expand description
Split source code into tokens.
- Strips line comments (
//and#) - Splits on whitespace
- Splits tokens further on punctuation boundaries (keeps punctuation as separate tokens)
- Skips empty tokens