Function lex 

pub fn lex<L: TokenSource>(
    input: impl AsRef<str>,
    lexer: impl FnOnce(InputStream) -> L,
) -> CommonTokenStream<L>

Lexes UTF-8 text into an eagerly filled token stream without constructing a parser.

Pass the generated lexer constructor, for example lex(src, MyGrammarLexer::new).

The stream retains every emitted token, including EOF and tokens on hidden or custom channels. Lexer rules using skip do not emit tokens.

With use antlr4_runtime::Token as _; and the generated module’s metadata(), print each token’s vocabulary name, numeric channel, and text: let vocabulary = metadata().vocabulary(); for token in lex(src, MyGrammarLexer::new).tokens() { println!("type={} channel={} text={:?}", vocabulary.display_name(token.token_type()), token.channel(), token.text()); }

number_of_source_errors() reports buffered lexer diagnostics. After iterating tokens(), drain_source_errors() retrieves those diagnostics.

Panics

Panics if buffering returns an crate::TokenStoreError. Construct the lexer and call CommonTokenStream::try_new to handle that error instead.