Function lamcal::parser::tokenize

pub fn tokenize(
    input: impl IntoIterator<Item = char>
) -> Result<Vec<(Token, CharPosition)>, ParseError>

Converts a stream of chars into a list of Tokens.

The returned tokens are paired with their position within the input stream. Hence this function actually converts the input stream into a Vec<(Token, CharPosition). The position information is used for ParseErrors found by the parse function.