Function bracket::lexer::collect

pub fn collect(s: &str, normalized: bool) -> Vec<Token>

Collect the input source into a vector of tokens.

If the normalized flag is given consecutive text tokens are coalesced into a single token.

The normalized flag is useful for test cases; the parser will perform it's own normalization to reduce the number of passes on the token stream.