Struct TermLexer 

pub struct TermLexer<I>
where
    I: TryNextWithContext<Arena, Item = u8, Error: Display + 'static>,
{ /* private fields */ }

The lexer for Prolog-like terms.

TermLexer tokenizes input stream into TermTokens using DFA tables generated by parlex-gen’s alex tool. It maintains lexer state, manages nested constructs, and recognizes operators defined in OperDefs.

TermLexer<I> adapts a byte-oriented input stream I (that supports contextual access to an Arena) into an iterator-like interface that yields TermTokens. Internally, it owns a lower-level Lexer driven by TermLexerDriver, which handles rule actions (e.g., interning terms, parsing numbers, skipping comments/whitespace).

The generic parameter I must implement TryNextWithContext<Item = u8, Context = Arena>, allowing the lexer to pull bytes and intern terms while tokenizing.

Type Parameters

• I: The input source implementing TryNextWithContext<Arena> over bytes.

Output

Each successful step yields a TermToken, which carries:

• a token kind (TokenID),
• an optional payload (Value),
• a 1-based line number (line_no),
• an optional index into operator definition table.

Errors

Methods return a [LexerError<I::Error, TermParseError>], where:

• I::Error is any error produced by the underlying input,
• [TermParseError] covers lexical/parsing/UTF-8/term errors.

Example

let mut arena = Arena::new();
let input = IterInput::from("hello\n +\n world\n\n123".bytes());
let mut lexer = TermLexer::try_new(input).unwrap();
let vs = lexer.try_collect_with_context(&mut arena).unwrap();
assert_eq!(vs.len(), 5);

Implementations

impl<I> TermLexer<I>

where I: TryNextWithContext<Arena, Item = u8, Error: Display + 'static>,

pub fn try_new(input: I) -> Result<Self, ParlexError>

Parameters

• input: The input byte stream to be lexed.
• opers: Optional operator definitions ([OperDefs]) used to recognize operator tokens by fixity and precedence. If None, an empty operator table is created.

Returns

A ready-to-use TermLexer instance, or an error if the underlying [LexerCtx] initialization fails.

Errors

Returns an error if DFA table deserialization in [LexerCtx::try_new] fails or the input cannot be processed. Constructs a new term lexer from the given input stream and optional operator definition table.

This initializes an internal Lexer with a TermLexerDriver that performs rule actions such as:

• interning identifiers into the provided Arena (via context),
• converting matched byte slices into numbers/idents,
• tracking line numbers and comment nesting.

Parameters

• input: The input byte stream to be lexed.

Returns

A ready-to-use TermLexer instance, or an error if the underlying initializations failed.

Errors

Returns a [LexerError] if the lexer cannot be constructed from the given input and operatir table.

Trait Implementations

impl<I> TryNextWithContext<Arena, LexerStats> for TermLexer<I>

where I: TryNextWithContext<Arena, Item = u8, Error: Display + 'static>,

type Item = TermToken

Tokens produced by this lexer.

type Error = ParlexError

Unified error type.

fn try_next_with_context( &mut self, context: &mut Arena, ) -> Result<Option<TermToken>, ParlexError>

Advances the lexer and returns the next token, or None at end of input.

The provided context (an Arena) may be mutated by rule actions (for example, to intern terms). This method is fallible; both input and lexical errors are converted into Self::Error.

End of Input

When the lexer reaches the end of the input stream, it will typically emit a final TokenID::End token before returning None.

This explicit End token is expected by the Parlex parser to signal successful termination of a complete parsing unit. Consumers should treat this token as a logical end-of-sentence or end-of-expression marker, depending on the grammar.

If the input contains multiple independent sentences or expressions, the lexer may emit multiple End tokens—one after each completed unit. In such cases, the parser can restart or resume parsing after each End to produce multiple parse results from a single input stream.

Once all input has been consumed, the lexer returns None.

fn stats(&self) -> LexerStats

fn try_collect_with_context( &mut self, context: &mut C, ) -> Result<Vec<Self::Item>, Self::Error>

Collects all remaining items into a Vec, using the given context.