Function tokenize 

pub fn tokenize(format: &str, source: &str, mode: Mode) -> Vec<Token>

Tokenize source code in the given format with the given mode. Returns a Vec. Never panics on empty input — returns empty Vec.

This is the display/reporter path. For the detection path, use tokenize_to_detection.