Struct TermLexer 

pub struct TermLexer<I>
where
    I: FusedIterator<Item = u8>,
{
    pub opers: OperDefs,
    /* private fields */
}

The main lexer for Prolog-like terms.

TermLexer tokenizes input streams into TermTokens using DFA tables generated by parlex-gen’s alex tool. It maintains lexer state, manages nested constructs, and recognizes operators defined in OperDefs.

Type Parameters

• I: The input source implementing FusedIterator over bytes.

Fields

opers: OperDefs

The operator definition table used to resolve operator fixity, precedence, and associativity during lexing.

Implementations

impl<I> TermLexer<I>

where I: FusedIterator<Item = u8>,

Implementation of TermLexer methods.

This impl provides core construction logic for initializing a new term lexer instance. It prepares the internal LexerCtx state, sets up operator definitions, and initializes all internal counters used for managing nested structures and special tokens.

Type Parameters

• I: The input source, which must implement FusedIterator over bytes.

pub fn try_new(input: I, opers: Option<OperDefs>) -> Result<Self>

Constructs a new TermLexer from the given input stream.

Initializes the internal LexerCtx, loads operator definitions if provided, and resets all nesting and parsing counters.

Parameters

• input: The input byte stream to be lexed.
• opers: Optional operator definitions (OperDefs) used to recognize operator tokens by fixity and precedence. If None, an empty operator table is created.

Returns

A ready-to-use TermLexer instance, or an error if the underlying LexerCtx initialization fails.

Errors

Returns an error if DFA table deserialization in LexerCtx::try_new fails or the input cannot be processed.

Trait Implementations

impl<I> Lexer<Arena> for TermLexer<I>

where I: FusedIterator<Item = u8>,

Implements the Lexer trait for TermLexer, integrating with the parlex core library.

This binding wires the generated DFA / rule set (LexData) to the concrete term-lexing behavior provided by TermLexer. It exposes the lexer context and defines the rule action callback that builds TermTokens.

Associated Types

• Input — The input byte iterator (must be FusedIterator<Item = u8>).
• LexerData — The generated lexer tables and rule enums (LexData).
• Token — The token type produced by this lexer (TermToken).

fn ctx(&self) -> &LexerCtx<Self::Input, Self::LexerData, Self::Token>

Returns a shared reference to the internal LexerCtx.

fn ctx_mut( &mut self, ) -> &mut LexerCtx<Self::Input, Self::LexerData, Self::Token>

Returns a mutable reference to the internal LexerCtx.

fn action( &mut self, arena: &mut Arena, rule: <Self::LexerData as LexerData>::LexerRule, ) -> Result<()>

The primary user callback invoked by the parlex lexer for each matched rule.

This method implements the term-specific lexing logic:

• constructs arena-backed values (atoms, numbers, strings, dates, etc.),
• tracks nesting (parentheses, braces, comments, script blocks),
• consults OperDefs to recognize operators and annotate tokens with operator indices,
• and emits tokens via the yield_* helpers.

Parameters

• arena: The Arena used for allocating term values produced during lexing.
• rule: The matched lexer rule (from the generated LexerData::LexerRule).

Returns

Ok(()) on success; an error if token construction or state handling fails.

Errors

Propagates errors from value parsing (e.g., numeric/date parsing), arena allocation, or invalid state transitions.

type Input = I

The input stream type producing bytes for lexing.

type LexerData = LexData

The lexer data providing DFA tables and rule definitions. Must be provided by the implementer and must be the struct generated by parlex-gen’s lexer generator, Alex.

type Token = TermToken

The token type produced by the lexer. Must be provided by the implementor.

fn stats(&self) -> LexerStats

Returns current lexer statistics.

fn try_collect(&mut self, user_data: &mut U) -> Result<Vec<Self::Token>, Error>

Collects all tokens from the input until exhaustion.

fn try_next(&mut self, user_data: &mut U) -> Result<Option<Self::Token>, Error>

Attempts to fetch the next token from the input stream.

fn accum(&mut self)

Switches on accumulation of bytes into the buffer.

fn begin(&mut self, mode: <Self::LexerData as LexerData>::LexerMode)

Switches the lexer to a different mode.

fn yield_token(&mut self, token: Self::Token)

Emits a token into the output queue.

fn clear(&mut self)

Clears the current buffer and disables accumulation.

fn take_bytes(&mut self) -> Vec<u8>

Takes accumulated bytes from the main buffer.

fn take_bytes2(&mut self) -> Vec<u8>

Takes accumulated bytes from the secondary buffer.

fn take_str(&mut self) -> Result<SmartString<LazyCompact>, Error>

Takes accumulated bytes from the main buffer and converts them into a UTF-8 string.

fn take_str2(&mut self) -> Result<SmartString<LazyCompact>, Error>

Takes accumulated bytes from the secondary buffer and converts them into a UTF-8 string.

fn extend_buffer2_with_buffer(&mut self)

Extends the secondary buffer with the contents of the main buffer.