tokens_in_line

token_iter

Function tokens_in_line 

Source 
pub fn tokens_in_line<'a, Str, Token: 'a, Tokenizer>(
    line: Str,
    tokenizer: &'a Tokenizer,
) -> impl Iterator<Item = (Range<usize>, Token)> + 'a
where
    Str: Into<&'a str>,
    Tokenizer: Fn(&mut Lexer<'a>) -> Option<Token>,
Expand description

Returns an Iterator over the tokens found in the specified line, along with their column ranges. Column numbers start at 0.