pub fn tokenize_code(source: &str, language_name: &str) -> Vec<Vec<CodeToken>>Expand description
The source tokenized per LINE: each line’s tokens concatenate to exactly
its text (no \n), adjacent tokens of one kind merged; an empty line is
an empty list. An unknown language tokenizes as plain.