Skip to main content

token_stream

Function token_stream 

Source
pub fn token_stream<'t, 's: 't>(
    source: &'s str,
    tokens: &'t [(Token<'s>, SimpleSpan)],
) -> impl TokenInput<'t, 's>
Expand description

Turn a source string and its lexed tokens into a parser input.

The end-of-input span is source.len()..source.len() so that an error at the end of the input still carries a byte offset the diagnostics layer can translate.

use temps_core::{common::token_stream, lexer::lex};

let input = "in 5 minutes";
let tokens = lex(input);
let stream = token_stream(input, &tokens);