Function tokenizer_lib::StreamedTokenChannel::new

pub fn new<T: PartialEq, TData>(
) -> (StreamedTokenSender<T, TData>, StreamedTokenReader<T, TData>)

Will return a TokenSender and TokenReader for use when lexing and parsing in separate threads Unlike StaticTokenChannel it does not buffer all the tokens before parsing can begin