Struct lib_ruby_parser::Lexer
source · pub struct Lexer {
pub lex_state: LexState,
pub cond: StackState,
pub cmdarg: StackState,
pub static_env: StaticEnvironment,
/* private fields */
}Expand description
A struct responsible for converting a given input into a sequence of tokens
Fields§
§lex_state: LexStateCurrent state of the lexer, used internally for testing
cond: StackStateInternal field, used to differentiate kDO_COND vs kDO, exposed for internal testing
cmdarg: StackStateInternal field, used to differentiate kDO_BLOCK vs kDO, exposed for internal testing
static_env: StaticEnvironmentStack of sets of variables in current scopes. Each stack item represents locals in the scope.
You can use it to pre-define some locals and parse your input as if these locals exist.
For example, you can parse the following code
a = b + c
as
Send(LocalVar(a), "+", LocalVar(b))
by declaring a and b as locals using
parser.lexer.static_env.declare("a")
parser.lexer.static_env.declare("b")
parser.parse()
Implementations§
source§impl Lexer
impl Lexer
sourcepub fn new<Bytes, Name>(
bytes: Bytes,
name: Name,
decoder: Option<Decoder>
) -> Self
pub fn new<Bytes, Name>( bytes: Bytes, name: Name, decoder: Option<Decoder> ) -> Self
Constructs an instance of Lexer
sourcepub fn tokenize_until_eof(&mut self) -> Vec<Token>
pub fn tokenize_until_eof(&mut self) -> Vec<Token>
Tokenizes given input until EOF
Keep in mind that Lexer in Ruby is driven by Parser, and so this method on its own can return a wrong sequence of tokens. It’s used internally to test simple inputs.
If you need to get tokens better use ParserResult::tokens field
source§impl Lexer
impl Lexer
sourcepub const END_OF_INPUT: i32 = 0i32
pub const END_OF_INPUT: i32 = 0i32
Token "end-of-input", to be returned by the scanner.
sourcepub const kDO_LAMBDA: i32 = 284i32
pub const kDO_LAMBDA: i32 = 284i32
Token "`do' for lambda", to be returned by the scanner.
sourcepub const kUNLESS_MOD: i32 = 296i32
pub const kUNLESS_MOD: i32 = 296i32
Token "`unless' modifier", to be returned by the scanner.
sourcepub const kWHILE_MOD: i32 = 297i32
pub const kWHILE_MOD: i32 = 297i32
Token "`while' modifier", to be returned by the scanner.
sourcepub const kUNTIL_MOD: i32 = 298i32
pub const kUNTIL_MOD: i32 = 298i32
Token "`until' modifier", to be returned by the scanner.
sourcepub const kRESCUE_MOD: i32 = 299i32
pub const kRESCUE_MOD: i32 = 299i32
Token "`rescue' modifier", to be returned by the scanner.
sourcepub const k__ENCODING__: i32 = 306i32
pub const k__ENCODING__: i32 = 306i32
Token "`__ENCODING__'", to be returned by the scanner.
sourcepub const tIDENTIFIER: i32 = 307i32
pub const tIDENTIFIER: i32 = 307i32
Token "local variable or method", to be returned by the scanner.
sourcepub const tIMAGINARY: i32 = 317i32
pub const tIMAGINARY: i32 = 317i32
Token "imaginary literal", to be returned by the scanner.
sourcepub const tSTRING_CONTENT: i32 = 321i32
pub const tSTRING_CONTENT: i32 = 321i32
Token "literal content", to be returned by the scanner.
sourcepub const tREGEXP_END: i32 = 322i32
pub const tREGEXP_END: i32 = 322i32
Token tREGEXP_END, to be returned by the scanner.
sourcepub const tBACKSLASH: i32 = 324i32
pub const tBACKSLASH: i32 = 324i32
Token "backslash", to be returned by the scanner.
sourcepub const tSLASH_T: i32 = 326i32
pub const tSLASH_T: i32 = 326i32
Token "escaped horizontal tab", to be returned by the scanner.
sourcepub const tSLASH_R: i32 = 328i32
pub const tSLASH_R: i32 = 328i32
Token "escaped carriage return", to be returned by the scanner.
sourcepub const tLPAREN_ARG: i32 = 357i32
pub const tLPAREN_ARG: i32 = 357i32
Token "( arg", to be returned by the scanner.
sourcepub const tLBRACE_ARG: i32 = 361i32
pub const tLBRACE_ARG: i32 = 361i32
Token "{ arg", to be returned by the scanner.
sourcepub const tSTRING_BEG: i32 = 367i32
pub const tSTRING_BEG: i32 = 367i32
Token "string begin", to be returned by the scanner.
sourcepub const tXSTRING_BEG: i32 = 368i32
pub const tXSTRING_BEG: i32 = 368i32
Token "backtick literal", to be returned by the scanner.
sourcepub const tREGEXP_BEG: i32 = 369i32
pub const tREGEXP_BEG: i32 = 369i32
Token "regexp literal", to be returned by the scanner.
sourcepub const tWORDS_BEG: i32 = 370i32
pub const tWORDS_BEG: i32 = 370i32
Token "word list", to be returned by the scanner.
sourcepub const tQWORDS_BEG: i32 = 371i32
pub const tQWORDS_BEG: i32 = 371i32
Token "verbatim word list", to be returned by the scanner.
sourcepub const tSYMBOLS_BEG: i32 = 372i32
pub const tSYMBOLS_BEG: i32 = 372i32
Token "symbol list", to be returned by the scanner.
sourcepub const tQSYMBOLS_BEG: i32 = 373i32
pub const tQSYMBOLS_BEG: i32 = 373i32
Token "verbatim symbol list", to be returned by the scanner.
sourcepub const tSTRING_END: i32 = 374i32
pub const tSTRING_END: i32 = 374i32
Token "string end", to be returned by the scanner.
sourcepub const tSTRING_DEND: i32 = 375i32
pub const tSTRING_DEND: i32 = 375i32
Token "tRCURLY", to be returned by the scanner.
sourcepub const tSTRING_DBEG: i32 = 376i32
pub const tSTRING_DBEG: i32 = 376i32
Token tSTRING_DBEG, to be returned by the scanner.
sourcepub const tSTRING_DVAR: i32 = 377i32
pub const tSTRING_DVAR: i32 = 377i32
Token tSTRING_DVAR, to be returned by the scanner.
sourcepub const tLABEL_END: i32 = 379i32
pub const tLABEL_END: i32 = 379i32
Token tLABEL_END, to be returned by the scanner.
sourcepub const tBACK_REF2: i32 = 389i32
pub const tBACK_REF2: i32 = 389i32
Token "`", to be returned by the scanner.
sourcepub const tUMINUS_NUM: i32 = 406i32
pub const tUMINUS_NUM: i32 = 406i32
Token tUMINUS_NUM, to be returned by the scanner.
sourcepub const tLAST_TOKEN: i32 = 407i32
pub const tLAST_TOKEN: i32 = 407i32
Token tLAST_TOKEN, to be returned by the scanner.