pub struct Lexer {
pub lex_state: LexState,
pub cond: StackState,
pub cmdarg: StackState,
pub static_env: StaticEnvironment,
// some fields omitted
}Expand description
A struct responsible for converting a given input
into a sequence of tokens
Current state of the lexer, used internally for testing
Internal field, used to differentiate kDO_COND vs kDO,
exposed for internal testing
Internal field, used to differentiate kDO_BLOCK vs kDO,
exposed for internal testing
Stack of sets of variables in current scopes.
Each stack item represents locals in the scope.
You can use it to pre-define some locals and parse
your input as if these locals exist.
For example, you can parse the following code
a = b + c
as
Send(LocalVar(a), "+", LocalVar(b))
by declaring a and b as locals using
parser.lexer.static_env.declare("a")
parser.lexer.static_env.declare("b")
parser.parse()
Constructs an instance of Lexer
Enables printing additional debugging during lexing
Tokenizes given input until EOF
Keep in mind that Lexer in Ruby is driven by Parser,
and so this method on its own can return a wrong sequence
of tokens. It’s used internally to test simple inputs.
If you need to get tokens better use ParserResult::tokens field
Token "end-of-input", to be returned by the scanner.
Token error, to be returned by the scanner.
Token "invalid token", to be returned by the scanner.
Token "`class'", to be returned by the scanner.
Token "`module'", to be returned by the scanner.
Token "`def'", to be returned by the scanner.
Token "`undef'", to be returned by the scanner.
Token "`begin'", to be returned by the scanner.
Token "`rescue'", to be returned by the scanner.
Token "`ensure'", to be returned by the scanner.
Token "`end'", to be returned by the scanner.
Token "`if'", to be returned by the scanner.
Token "`unless'", to be returned by the scanner.
Token "`then'", to be returned by the scanner.
Token "`elsif'", to be returned by the scanner.
Token "`else'", to be returned by the scanner.
Token "`case'", to be returned by the scanner.
Token "`when'", to be returned by the scanner.
Token "`while'", to be returned by the scanner.
Token "`until'", to be returned by the scanner.
Token "`for'", to be returned by the scanner.
Token "`break'", to be returned by the scanner.
Token "`next'", to be returned by the scanner.
Token "`redo'", to be returned by the scanner.
Token "`retry'", to be returned by the scanner.
Token "`in'", to be returned by the scanner.
Token "`do'", to be returned by the scanner.
Token "`do' for condition", to be returned by the scanner.
Token "`do' for block", to be returned by the scanner.
Token "`do' for lambda", to be returned by the scanner.
Token "`return'", to be returned by the scanner.
Token "`yield'", to be returned by the scanner.
Token "`super'", to be returned by the scanner.
Token "`self'", to be returned by the scanner.
Token "`nil'", to be returned by the scanner.
Token "`true'", to be returned by the scanner.
Token "`false'", to be returned by the scanner.
Token "`and'", to be returned by the scanner.
Token "`or'", to be returned by the scanner.
Token "`not'", to be returned by the scanner.
Token "`if' modifier", to be returned by the scanner.
Token "`unless' modifier", to be returned by the scanner.
Token "`while' modifier", to be returned by the scanner.
Token "`until' modifier", to be returned by the scanner.
Token "`rescue' modifier", to be returned by the scanner.
Token "`alias'", to be returned by the scanner.
Token "`defined?'", to be returned by the scanner.
Token "`BEGIN'", to be returned by the scanner.
Token "`END'", to be returned by the scanner.
Token "`__LINE__'", to be returned by the scanner.
Token "`__FILE__'", to be returned by the scanner.
Token "`__ENCODING__'", to be returned by the scanner.
Token "local variable or method", to be returned by the scanner.
Token "method", to be returned by the scanner.
Token "global variable", to be returned by the scanner.
Token "instance variable", to be returned by the scanner.
Token "constant", to be returned by the scanner.
Token "class variable", to be returned by the scanner.
Token "label", to be returned by the scanner.
Token "integer literal", to be returned by the scanner.
Token "float literal", to be returned by the scanner.
Token "rational literal", to be returned by the scanner.
Token "imaginary literal", to be returned by the scanner.
Token "char literal", to be returned by the scanner.
Token "numbered reference", to be returned by the scanner.
Token "back reference", to be returned by the scanner.
Token "literal content", to be returned by the scanner.
Token tREGEXP_END, to be returned by the scanner.
Token tDOT, to be returned by the scanner.
Token "backslash", to be returned by the scanner.
Token "escaped space", to be returned by the scanner.
Token "escaped horizontal tab", to be returned by the scanner.
Token "escaped form feed", to be returned by the scanner.
Token "escaped carriage return", to be returned by the scanner.
Token "escaped vertical tab", to be returned by the scanner.
Token "unary+", to be returned by the scanner.
Token "unary-", to be returned by the scanner.
Token "**", to be returned by the scanner.
Token "<=>", to be returned by the scanner.
Token "==", to be returned by the scanner.
Token "===", to be returned by the scanner.
Token "!=", to be returned by the scanner.
Token ">=", to be returned by the scanner.
Token "<=", to be returned by the scanner.
Token "&&", to be returned by the scanner.
Token "||", to be returned by the scanner.
Token "=~", to be returned by the scanner.
Token "!~", to be returned by the scanner.
Token "..", to be returned by the scanner.
Token "...", to be returned by the scanner.
Token "(..", to be returned by the scanner.
Token "(...", to be returned by the scanner.
Token "[]", to be returned by the scanner.
Token "[]=", to be returned by the scanner.
Token "<<", to be returned by the scanner.
Token ">>", to be returned by the scanner.
Token "&.", to be returned by the scanner.
Token "::", to be returned by the scanner.
Token ":: at EXPR_BEG", to be returned by the scanner.
Token "operator-assignment", to be returned by the scanner.
Token "=>", to be returned by the scanner.
Token "(", to be returned by the scanner.
Token "( arg", to be returned by the scanner.
Token ")", to be returned by the scanner.
Token "[", to be returned by the scanner.
Token "{", to be returned by the scanner.
Token "{ arg", to be returned by the scanner.
Token "*", to be returned by the scanner.
Token "**arg", to be returned by the scanner.
Token "&", to be returned by the scanner.
Token "->", to be returned by the scanner.
Token "symbol literal", to be returned by the scanner.
Token "string begin", to be returned by the scanner.
Token "backtick literal", to be returned by the scanner.
Token "regexp literal", to be returned by the scanner.
Token "word list", to be returned by the scanner.
Token "verbatim word list", to be returned by the scanner.
Token "symbol list", to be returned by the scanner.
Token "verbatim symbol list", to be returned by the scanner.
Token "string end", to be returned by the scanner.
Token "tRCURLY", to be returned by the scanner.
Token tSTRING_DBEG, to be returned by the scanner.
Token tSTRING_DVAR, to be returned by the scanner.
Token tLAMBEG, to be returned by the scanner.
Token tLABEL_END, to be returned by the scanner.
Token ",", to be returned by the scanner.
Token "{ (tLCURLY)", to be returned by the scanner.
Token "}", to be returned by the scanner.
Token "[ (tLBRACK2)", to be returned by the scanner.
Token "=", to be returned by the scanner.
Token "|", to be returned by the scanner.
Token "& (tAMPER2)", to be returned by the scanner.
Token ">", to be returned by the scanner.
Token "<", to be returned by the scanner.
Token "`", to be returned by the scanner.
Token "^", to be returned by the scanner.
Token "( (tLPAREN2)", to be returned by the scanner.
Token "]", to be returned by the scanner.
Token ";", to be returned by the scanner.
Token " ", to be returned by the scanner.
Token "\n", to be returned by the scanner.
Token "+", to be returned by the scanner.
Token "-", to be returned by the scanner.
Token "* (tSTAR2)", to be returned by the scanner.
Token "/", to be returned by the scanner.
Token "%", to be returned by the scanner.
Token "~", to be returned by the scanner.
Token "!", to be returned by the scanner.
Token tLOWEST, to be returned by the scanner.
Token tEH, to be returned by the scanner.
Token tCOLON, to be returned by the scanner.
Token tUMINUS_NUM, to be returned by the scanner.
Token tLAST_TOKEN, to be returned by the scanner.
Formats the value using the given formatter. Read more
Returns the “default value” for a type. Read more
Immutably borrows from an owned value. Read more
Mutably borrows from an owned value. Read more
The type returned in the event of a conversion error.
The type returned in the event of a conversion error.