Function normalize 

pub fn normalize(code: &str) -> Vec<Token>

Normalizes source code into a token stream for duplicate detection

Normalization Rules

• Comments are ignored
• Whitespace is ignored
• Identifiers → $$ID
• String literals → $$STR
• Number literals → $$NUM
• Keywords are preserved

Arguments

• code - The source code to normalize

Returns

• Vec<Token> - Normalized token stream