pub struct Lexer {
pub lex_state: LexState,
pub cond: StackState,
pub cmdarg: StackState,
pub static_env: StaticEnvironment,
// some fields omitted
}
Expand description
A struct responsible for converting a given input
into a sequence of tokens
Current state of the lexer, used internally for testing
Internal field, used to differentiate kDO_COND vs kDO,
exposed for internal testing
Internal field, used to differentiate kDO_BLOCK vs kDO,
exposed for internal testing
Stack of sets of variables in current scopes.
Each stack item represents locals in the scope.
You can use it to pre-define some locals and parse
your input as if these locals exist.
For example, you can parse the following code
as
Send(LocalVar(a), "+", LocalVar(b))
by declaring a
and b
as locals using
parser.lexer.static_env.declare("a")
parser.lexer.static_env.declare("b")
parser.parse()
Constructs an instance of Lexer
Tokenizes given input until EOF
Keep in mind that Lexer in Ruby is driven by Parser,
and so this method on its own can return a wrong sequence
of tokens. It’s used internally to test simple inputs.
If you need to get tokens better use ParserResult::tokens
field
Token "end-of-input"
, to be returned by the scanner.
Token error
, to be returned by the scanner.
Token "invalid token"
, to be returned by the scanner.
Token "`class'"
, to be returned by the scanner.
Token "`module'"
, to be returned by the scanner.
Token "`def'"
, to be returned by the scanner.
Token "`undef'"
, to be returned by the scanner.
Token "`begin'"
, to be returned by the scanner.
Token "`rescue'"
, to be returned by the scanner.
Token "`ensure'"
, to be returned by the scanner.
Token "`end'"
, to be returned by the scanner.
Token "`if'"
, to be returned by the scanner.
Token "`unless'"
, to be returned by the scanner.
Token "`then'"
, to be returned by the scanner.
Token "`elsif'"
, to be returned by the scanner.
Token "`else'"
, to be returned by the scanner.
Token "`case'"
, to be returned by the scanner.
Token "`when'"
, to be returned by the scanner.
Token "`while'"
, to be returned by the scanner.
Token "`until'"
, to be returned by the scanner.
Token "`for'"
, to be returned by the scanner.
Token "`break'"
, to be returned by the scanner.
Token "`next'"
, to be returned by the scanner.
Token "`redo'"
, to be returned by the scanner.
Token "`retry'"
, to be returned by the scanner.
Token "`in'"
, to be returned by the scanner.
Token "`do'"
, to be returned by the scanner.
Token "`do' for condition"
, to be returned by the scanner.
Token "`do' for block"
, to be returned by the scanner.
Token "`do' for lambda"
, to be returned by the scanner.
Token "`return'"
, to be returned by the scanner.
Token "`yield'"
, to be returned by the scanner.
Token "`super'"
, to be returned by the scanner.
Token "`self'"
, to be returned by the scanner.
Token "`nil'"
, to be returned by the scanner.
Token "`true'"
, to be returned by the scanner.
Token "`false'"
, to be returned by the scanner.
Token "`and'"
, to be returned by the scanner.
Token "`or'"
, to be returned by the scanner.
Token "`not'"
, to be returned by the scanner.
Token "`if' modifier"
, to be returned by the scanner.
Token "`unless' modifier"
, to be returned by the scanner.
Token "`while' modifier"
, to be returned by the scanner.
Token "`until' modifier"
, to be returned by the scanner.
Token "`rescue' modifier"
, to be returned by the scanner.
Token "`alias'"
, to be returned by the scanner.
Token "`defined?'"
, to be returned by the scanner.
Token "`BEGIN'"
, to be returned by the scanner.
Token "`END'"
, to be returned by the scanner.
Token "`__LINE__'"
, to be returned by the scanner.
Token "`__FILE__'"
, to be returned by the scanner.
Token "`__ENCODING__'"
, to be returned by the scanner.
Token "local variable or method"
, to be returned by the scanner.
Token "method"
, to be returned by the scanner.
Token "global variable"
, to be returned by the scanner.
Token "instance variable"
, to be returned by the scanner.
Token "constant"
, to be returned by the scanner.
Token "class variable"
, to be returned by the scanner.
Token "label"
, to be returned by the scanner.
Token "integer literal"
, to be returned by the scanner.
Token "float literal"
, to be returned by the scanner.
Token "rational literal"
, to be returned by the scanner.
Token "imaginary literal"
, to be returned by the scanner.
Token "char literal"
, to be returned by the scanner.
Token "numbered reference"
, to be returned by the scanner.
Token "back reference"
, to be returned by the scanner.
Token "literal content"
, to be returned by the scanner.
Token tREGEXP_END
, to be returned by the scanner.
Token tDOT
, to be returned by the scanner.
Token "backslash"
, to be returned by the scanner.
Token "escaped space"
, to be returned by the scanner.
Token "escaped horizontal tab"
, to be returned by the scanner.
Token "escaped form feed"
, to be returned by the scanner.
Token "escaped carriage return"
, to be returned by the scanner.
Token "escaped vertical tab"
, to be returned by the scanner.
Token "unary+"
, to be returned by the scanner.
Token "unary-"
, to be returned by the scanner.
Token "**"
, to be returned by the scanner.
Token "<=>"
, to be returned by the scanner.
Token "=="
, to be returned by the scanner.
Token "==="
, to be returned by the scanner.
Token "!="
, to be returned by the scanner.
Token ">="
, to be returned by the scanner.
Token "<="
, to be returned by the scanner.
Token "&&"
, to be returned by the scanner.
Token "||"
, to be returned by the scanner.
Token "=~"
, to be returned by the scanner.
Token "!~"
, to be returned by the scanner.
Token ".."
, to be returned by the scanner.
Token "..."
, to be returned by the scanner.
Token "(.."
, to be returned by the scanner.
Token "(..."
, to be returned by the scanner.
Token "[]"
, to be returned by the scanner.
Token "[]="
, to be returned by the scanner.
Token "<<"
, to be returned by the scanner.
Token ">>"
, to be returned by the scanner.
Token "&."
, to be returned by the scanner.
Token "::"
, to be returned by the scanner.
Token ":: at EXPR_BEG"
, to be returned by the scanner.
Token "operator-assignment"
, to be returned by the scanner.
Token "=>"
, to be returned by the scanner.
Token "("
, to be returned by the scanner.
Token "( arg"
, to be returned by the scanner.
Token ")"
, to be returned by the scanner.
Token "["
, to be returned by the scanner.
Token "{"
, to be returned by the scanner.
Token "{ arg"
, to be returned by the scanner.
Token "*"
, to be returned by the scanner.
Token "**arg"
, to be returned by the scanner.
Token "&"
, to be returned by the scanner.
Token "->"
, to be returned by the scanner.
Token "symbol literal"
, to be returned by the scanner.
Token "string begin"
, to be returned by the scanner.
Token "backtick literal"
, to be returned by the scanner.
Token "regexp literal"
, to be returned by the scanner.
Token "word list"
, to be returned by the scanner.
Token "verbatim word list"
, to be returned by the scanner.
Token "symbol list"
, to be returned by the scanner.
Token "verbatim symbol list"
, to be returned by the scanner.
Token "string end"
, to be returned by the scanner.
Token "tRCURLY"
, to be returned by the scanner.
Token tSTRING_DBEG
, to be returned by the scanner.
Token tSTRING_DVAR
, to be returned by the scanner.
Token tLAMBEG
, to be returned by the scanner.
Token tLABEL_END
, to be returned by the scanner.
Token ","
, to be returned by the scanner.
Token "{ (tLCURLY)"
, to be returned by the scanner.
Token "}"
, to be returned by the scanner.
Token "[ (tLBRACK2)"
, to be returned by the scanner.
Token "="
, to be returned by the scanner.
Token "|"
, to be returned by the scanner.
Token "& (tAMPER2)"
, to be returned by the scanner.
Token ">"
, to be returned by the scanner.
Token "<"
, to be returned by the scanner.
Token "`"
, to be returned by the scanner.
Token "^"
, to be returned by the scanner.
Token "( (tLPAREN2)"
, to be returned by the scanner.
Token "]"
, to be returned by the scanner.
Token ";"
, to be returned by the scanner.
Token " "
, to be returned by the scanner.
Token "\n"
, to be returned by the scanner.
Token "+"
, to be returned by the scanner.
Token "-"
, to be returned by the scanner.
Token "* (tSTAR2)"
, to be returned by the scanner.
Token "/"
, to be returned by the scanner.
Token "%"
, to be returned by the scanner.
Token "~"
, to be returned by the scanner.
Token "!"
, to be returned by the scanner.
Token tLOWEST
, to be returned by the scanner.
Token tEH
, to be returned by the scanner.
Token tCOLON
, to be returned by the scanner.
Token tUMINUS_NUM
, to be returned by the scanner.
Token tLAST_TOKEN
, to be returned by the scanner.
Formats the value using the given formatter. Read more
Returns the “default value” for a type. Read more
impl<T> Any for T where
T: 'static + ?Sized,
Immutably borrows from an owned value. Read more
Mutably borrows from an owned value. Read more
impl<T, U> Into<U> for T where
U: From<T>,
The type returned in the event of a conversion error.
The type returned in the event of a conversion error.