Function tokenizer_lib::get_streamed_token_channel

pub fn get_streamed_token_channel<T>(
) -> (StreamedTokenSender<T>, StreamedTokenReader<T>)

Will return a TokenSender and TokenReader for use when lexing and parsing in separate threads Unlike StaticTokenChannel it does not buffer all the tokens before parsing can begin