Struct LexerCtx 

pub struct LexerCtx<I, D, T>
where
    D: LexerData,
{
    pub mode: D::LexerMode,
    pub accum_flag: bool,
    pub buffer: Vec<u8>,
    pub buffer2: Vec<u8>,
    pub end_flag: bool,
    pub line_no: usize,
    /* private fields */
}

Core lexer execution context.

LexerCtx manages the entire runtime state of the lexer, including its input stream, DFA tables, buffers, and recognized tokens. It is responsible for coordinating lexing operations, tracking position and mode, and maintaining internal statistics.

Type Parameters

• I: The input source, typically an iterator over bytes.
• D: The lexer definition, which implements the LexerData trait. This type is usually generated by the parlex-gen lexer generator (Alex).
• T: The token type produced by the lexer.

Fields

mode: D::LexerMode

The current lexer mode, defined by the lexer data type D.

accum_flag: bool

Indicates whether the lexer is currently accumulating token text.

buffer: Vec<u8>

Primary buffer used to accumulate the characters of the current token.

buffer2: Vec<u8>

Secondary buffer available to the user, for example, for constructing more complex tokens.

end_flag: bool

Signals that the end of input has been reached.

line_no: usize

The current line number in the input, used for diagnostics.

Implementations

impl<I, D, T> LexerCtx<I, D, T>

where I: FusedIterator<Item = u8>, D: LexerData, T: Token,

Implementation of LexerCtx methods.

This impl defines the primary construction and matching logic for the lexer context. It provides methods to initialize a new lexer context instance, load DFA tables, and perform token matching over the input stream.

Type Parameters

• I: The input source, which must be a FusedIterator yielding bytes.
• D: The lexer definition type implementing the LexerData trait. This type is usually generated by the parlex-gen lexer generator (Alex).
• T: The token type implementing the Token trait.

pub fn try_new(input: I) -> Result<Self>

Constructs a new LexerCtx from the given input.

This initializes DFA tables from D::dfa_bytes(), sets the lexer to its starting mode, and prepares all buffers and counters for processing.

Parameters

• input: A fused iterator of bytes serving as the lexer’s input stream.

Returns

A fully initialized lexer context, or an error if the DFA tables cannot be deserialized.

Errors

Returns an error if any DFA fails to deserialize via dense::DFA::from_bytes.