Module hime_redist::lexers::automaton

Expand description

Module for lexers’ automata

Structs

Represents the automaton of a lexer Binary data structure of lexers: u32: number of entries in the states index table – states offset table each entry is of the form: u32: offset of the state from the beginning of the states table in number of u16 – states table

AutomatonState

Represents a state in the automaton of a lexer Binary data structure: u16: number of matched terminals u16: total number of transitions u16: number of non-cached transitions – matched terminals u16: context identifier u16: index of the matched terminal – cache: 256 entries u16: next state’s index for index of the entry – transitions each transition is of the form: u16: start of the range u16: end of the range u16: next state’s index

AutomatonTransition

Represents a transition in the automaton of a lexer A transition is matched by a range of UTF-16 code points Its target is a state in the automaton

MatchedTerminal

Represents the information of a terminal matched at the state of a lexer’s automaton

TokenMatch

Represents a match in the input

Constants

DEAD_STATE

Identifier of an invalid state in an automaton

Functions

run_dfa

Runs the lexer’s DFA to match a terminal in the input ahead