pub fn lex_pattern_chunk<L>(
text: &str,
make_lexer: impl FnOnce(InputStream) -> L,
) -> Result<Vec<TokenSpec>, ParseTreePatternError>where
L: TokenSource,Expand description
Runs a lexer over one chunk of pattern text and returns its token specs
(every non-EOF token, on its original channel), suitable for a
PatternLexer.
This is the bridge generated parsers use to satisfy PatternLexer from
their concrete lexer: make_lexer builds a fresh lexer over the chunk’s
InputStream, the tokens are buffered, and each
non-EOF token becomes a TokenSpec::explicit(type, text) carrying its
channel. Like ANTLR’s tokenize, hidden-channel tokens (whitespace,
comments) are preserved on their channel so the interpreter skips them the
same way it does during a normal parse. Positions are dropped because
pattern trees compare by type and text, not span.
§Errors
Returns ParseTreePatternError::Tokenization if the lexer reports a
tokenization error for the chunk.