pub fn tokenize(
src: &[u8],
file_start: BytePos,
opts: Options,
interner: &mut Interner,
) -> (Vec<PpToken>, Vec<Diagnostic>)Expand description
Scans src to the end and returns every preprocessing token and every complaint.
The end of file token is included, because everything downstream wants somewhere to point when a construct runs off the end of the file.