Struct lib_ruby_parser::Lexer

source · [−]
pub struct Lexer {
    pub lex_state: LexState,
    pub cond: StackState,
    pub cmdarg: StackState,
    pub static_env: StaticEnvironment,
    /* private fields */
}
Expand description

A struct responsible for converting a given input into a sequence of tokens

Fields

lex_state: LexState

Current state of the lexer, used internally for testing

cond: StackState

Internal field, used to differentiate kDO_COND vs kDO, exposed for internal testing

cmdarg: StackState

Internal field, used to differentiate kDO_BLOCK vs kDO, exposed for internal testing

static_env: StaticEnvironment

Stack of sets of variables in current scopes. Each stack item represents locals in the scope.

You can use it to pre-define some locals and parse your input as if these locals exist.

For example, you can parse the following code

a = b + c

as

Send(LocalVar(a), "+", LocalVar(b))

by declaring a and b as locals using

parser.lexer.static_env.declare("a")
parser.lexer.static_env.declare("b")
parser.parse()

Implementations

Constructs an instance of Lexer

Tokenizes given input until EOF

Keep in mind that Lexer in Ruby is driven by Parser, and so this method on its own can return a wrong sequence of tokens. It’s used internally to test simple inputs.

If you need to get tokens better use ParserResult::tokens field

Token "end-of-input", to be returned by the scanner.

Token error, to be returned by the scanner.

Token "invalid token", to be returned by the scanner.

Token "`class'", to be returned by the scanner.

Token "`module'", to be returned by the scanner.

Token "`def'", to be returned by the scanner.

Token "`undef'", to be returned by the scanner.

Token "`begin'", to be returned by the scanner.

Token "`rescue'", to be returned by the scanner.

Token "`ensure'", to be returned by the scanner.

Token "`end'", to be returned by the scanner.

Token "`if'", to be returned by the scanner.

Token "`unless'", to be returned by the scanner.

Token "`then'", to be returned by the scanner.

Token "`elsif'", to be returned by the scanner.

Token "`else'", to be returned by the scanner.

Token "`case'", to be returned by the scanner.

Token "`when'", to be returned by the scanner.

Token "`while'", to be returned by the scanner.

Token "`until'", to be returned by the scanner.

Token "`for'", to be returned by the scanner.

Token "`break'", to be returned by the scanner.

Token "`next'", to be returned by the scanner.

Token "`redo'", to be returned by the scanner.

Token "`retry'", to be returned by the scanner.

Token "`in'", to be returned by the scanner.

Token "`do'", to be returned by the scanner.

Token "`do' for condition", to be returned by the scanner.

Token "`do' for block", to be returned by the scanner.

Token "`do' for lambda", to be returned by the scanner.

Token "`return'", to be returned by the scanner.

Token "`yield'", to be returned by the scanner.

Token "`super'", to be returned by the scanner.

Token "`self'", to be returned by the scanner.

Token "`nil'", to be returned by the scanner.

Token "`true'", to be returned by the scanner.

Token "`false'", to be returned by the scanner.

Token "`and'", to be returned by the scanner.

Token "`or'", to be returned by the scanner.

Token "`not'", to be returned by the scanner.

Token "`if' modifier", to be returned by the scanner.

Token "`unless' modifier", to be returned by the scanner.

Token "`while' modifier", to be returned by the scanner.

Token "`until' modifier", to be returned by the scanner.

Token "`rescue' modifier", to be returned by the scanner.

Token "`alias'", to be returned by the scanner.

Token "`defined?'", to be returned by the scanner.

Token "`BEGIN'", to be returned by the scanner.

Token "`END'", to be returned by the scanner.

source

pub const k__LINE__: i32 = 304i32

Token "`__LINE__'", to be returned by the scanner.

source

pub const k__FILE__: i32 = 305i32

Token "`__FILE__'", to be returned by the scanner.

source

pub const k__ENCODING__: i32 = 306i32

Token "`__ENCODING__'", to be returned by the scanner.

Token "local variable or method", to be returned by the scanner.

Token "method", to be returned by the scanner.

Token "global variable", to be returned by the scanner.

Token "instance variable", to be returned by the scanner.

Token "constant", to be returned by the scanner.

Token "class variable", to be returned by the scanner.

Token "label", to be returned by the scanner.

Token "integer literal", to be returned by the scanner.

Token "float literal", to be returned by the scanner.

Token "rational literal", to be returned by the scanner.

Token "imaginary literal", to be returned by the scanner.

Token "char literal", to be returned by the scanner.

Token "numbered reference", to be returned by the scanner.

Token "back reference", to be returned by the scanner.

source

pub const tSTRING_CONTENT: i32 = 321i32

Token "literal content", to be returned by the scanner.

Token tREGEXP_END, to be returned by the scanner.

Token tDOT, to be returned by the scanner.

Token "backslash", to be returned by the scanner.

Token "escaped space", to be returned by the scanner.

Token "escaped horizontal tab", to be returned by the scanner.

Token "escaped form feed", to be returned by the scanner.

Token "escaped carriage return", to be returned by the scanner.

Token "escaped vertical tab", to be returned by the scanner.

Token "unary+", to be returned by the scanner.

Token "unary-", to be returned by the scanner.

Token "**", to be returned by the scanner.

Token "<=>", to be returned by the scanner.

Token "==", to be returned by the scanner.

Token "===", to be returned by the scanner.

Token "!=", to be returned by the scanner.

Token ">=", to be returned by the scanner.

Token "<=", to be returned by the scanner.

Token "&&", to be returned by the scanner.

Token "||", to be returned by the scanner.

Token "=~", to be returned by the scanner.

Token "!~", to be returned by the scanner.

Token "..", to be returned by the scanner.

Token "...", to be returned by the scanner.

Token "(..", to be returned by the scanner.

Token "(...", to be returned by the scanner.

Token "[]", to be returned by the scanner.

Token "[]=", to be returned by the scanner.

Token "<<", to be returned by the scanner.

Token ">>", to be returned by the scanner.

Token "&.", to be returned by the scanner.

Token "::", to be returned by the scanner.

Token ":: at EXPR_BEG", to be returned by the scanner.

Token "operator-assignment", to be returned by the scanner.

Token "=>", to be returned by the scanner.

Token "(", to be returned by the scanner.

Token "( arg", to be returned by the scanner.

Token ")", to be returned by the scanner.

Token "[", to be returned by the scanner.

Token "{", to be returned by the scanner.

Token "{ arg", to be returned by the scanner.

Token "*", to be returned by the scanner.

Token "**arg", to be returned by the scanner.

Token "&", to be returned by the scanner.

Token "->", to be returned by the scanner.

Token "symbol literal", to be returned by the scanner.

Token "string begin", to be returned by the scanner.

Token "backtick literal", to be returned by the scanner.

Token "regexp literal", to be returned by the scanner.

Token "word list", to be returned by the scanner.

Token "verbatim word list", to be returned by the scanner.

Token "symbol list", to be returned by the scanner.

Token "verbatim symbol list", to be returned by the scanner.

Token "string end", to be returned by the scanner.

Token "tRCURLY", to be returned by the scanner.

Token tSTRING_DBEG, to be returned by the scanner.

Token tSTRING_DVAR, to be returned by the scanner.

Token tLAMBEG, to be returned by the scanner.

Token tLABEL_END, to be returned by the scanner.

Token ",", to be returned by the scanner.

Token "{ (tLCURLY)", to be returned by the scanner.

Token "}", to be returned by the scanner.

Token "[ (tLBRACK2)", to be returned by the scanner.

Token "=", to be returned by the scanner.

Token "|", to be returned by the scanner.

Token "& (tAMPER2)", to be returned by the scanner.

Token ">", to be returned by the scanner.

Token "<", to be returned by the scanner.

Token "`", to be returned by the scanner.

Token "^", to be returned by the scanner.

Token "( (tLPAREN2)", to be returned by the scanner.

Token "]", to be returned by the scanner.

Token ";", to be returned by the scanner.

Token " ", to be returned by the scanner.

Token "\n", to be returned by the scanner.

Token "+", to be returned by the scanner.

Token "-", to be returned by the scanner.

Token "* (tSTAR2)", to be returned by the scanner.

Token "/", to be returned by the scanner.

Token "%", to be returned by the scanner.

Token "~", to be returned by the scanner.

Token "!", to be returned by the scanner.

Token tLOWEST, to be returned by the scanner.

Token tEH, to be returned by the scanner.

Token tCOLON, to be returned by the scanner.

Token tUMINUS_NUM, to be returned by the scanner.

Token tLAST_TOKEN, to be returned by the scanner.

Trait Implementations

Formats the value using the given formatter. Read more

Returns the “default value” for a type. Read more

Auto Trait Implementations

Blanket Implementations

Gets the TypeId of self. Read more

Immutably borrows from an owned value. Read more

Mutably borrows from an owned value. Read more

Returns the argument unchanged.

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

The type returned in the event of a conversion error.

Performs the conversion.

The type returned in the event of a conversion error.

Performs the conversion.