pub struct Lexer {
pub lex_state: LexState,
pub cond: StackState,
pub cmdarg: StackState,
pub static_env: StaticEnvironment,
/* private fields */
}Expand description
A struct responsible for converting a given input into a sequence of tokens
Fields§
§lex_state: LexStateCurrent state of the lexer, used internally for testing
cond: StackStateInternal field, used to differentiate kDO_COND vs kDO, exposed for internal testing
cmdarg: StackStateInternal field, used to differentiate kDO_BLOCK vs kDO, exposed for internal testing
static_env: StaticEnvironmentStack of sets of variables in current scopes. Each stack item represents locals in the scope.
You can use it to pre-define some locals and parse your input as if these locals exist.
For example, you can parse the following code
a = b + cas
Send(LocalVar(a), "+", LocalVar(b))by declaring a and b as locals using
parser.lexer.static_env.declare("a")
parser.lexer.static_env.declare("b")
parser.parse()Implementations§
Source§impl Lexer
impl Lexer
Sourcepub fn new<Bytes, Name>(
bytes: Bytes,
name: Name,
decoder: Option<Decoder>,
) -> Self
pub fn new<Bytes, Name>( bytes: Bytes, name: Name, decoder: Option<Decoder>, ) -> Self
Constructs an instance of Lexer
Sourcepub fn tokenize_until_eof(&mut self) -> Vec<Token>
pub fn tokenize_until_eof(&mut self) -> Vec<Token>
Tokenizes given input until EOF
Keep in mind that Lexer in Ruby is driven by Parser, and so this method on its own can return a wrong sequence of tokens. It’s used internally to test simple inputs.
If you need to get tokens better use ParserResult::tokens field
Source§impl Lexer
impl Lexer
Sourcepub const END_OF_INPUT: i32 = 0
pub const END_OF_INPUT: i32 = 0
Token "end-of-input", to be returned by the scanner.
Sourcepub const kDO_LAMBDA: i32 = 284
pub const kDO_LAMBDA: i32 = 284
Token "`do' for lambda", to be returned by the scanner.
Sourcepub const kUNLESS_MOD: i32 = 296
pub const kUNLESS_MOD: i32 = 296
Token "`unless' modifier", to be returned by the scanner.
Sourcepub const kWHILE_MOD: i32 = 297
pub const kWHILE_MOD: i32 = 297
Token "`while' modifier", to be returned by the scanner.
Sourcepub const kUNTIL_MOD: i32 = 298
pub const kUNTIL_MOD: i32 = 298
Token "`until' modifier", to be returned by the scanner.
Sourcepub const kRESCUE_MOD: i32 = 299
pub const kRESCUE_MOD: i32 = 299
Token "`rescue' modifier", to be returned by the scanner.
Sourcepub const k__ENCODING__: i32 = 306
pub const k__ENCODING__: i32 = 306
Token "`__ENCODING__'", to be returned by the scanner.
Sourcepub const tIDENTIFIER: i32 = 307
pub const tIDENTIFIER: i32 = 307
Token "local variable or method", to be returned by the scanner.
Sourcepub const tIMAGINARY: i32 = 317
pub const tIMAGINARY: i32 = 317
Token "imaginary literal", to be returned by the scanner.
Sourcepub const tSTRING_CONTENT: i32 = 321
pub const tSTRING_CONTENT: i32 = 321
Token "literal content", to be returned by the scanner.
Sourcepub const tREGEXP_END: i32 = 322
pub const tREGEXP_END: i32 = 322
Token tREGEXP_END, to be returned by the scanner.
Sourcepub const tBACKSLASH: i32 = 324
pub const tBACKSLASH: i32 = 324
Token "backslash", to be returned by the scanner.
Sourcepub const tLPAREN_ARG: i32 = 357
pub const tLPAREN_ARG: i32 = 357
Token "( arg", to be returned by the scanner.
Sourcepub const tLBRACE_ARG: i32 = 361
pub const tLBRACE_ARG: i32 = 361
Token "{ arg", to be returned by the scanner.
Sourcepub const tSTRING_BEG: i32 = 367
pub const tSTRING_BEG: i32 = 367
Token "string begin", to be returned by the scanner.
Sourcepub const tXSTRING_BEG: i32 = 368
pub const tXSTRING_BEG: i32 = 368
Token "backtick literal", to be returned by the scanner.
Sourcepub const tREGEXP_BEG: i32 = 369
pub const tREGEXP_BEG: i32 = 369
Token "regexp literal", to be returned by the scanner.
Sourcepub const tWORDS_BEG: i32 = 370
pub const tWORDS_BEG: i32 = 370
Token "word list", to be returned by the scanner.
Sourcepub const tQWORDS_BEG: i32 = 371
pub const tQWORDS_BEG: i32 = 371
Token "verbatim word list", to be returned by the scanner.
Sourcepub const tSYMBOLS_BEG: i32 = 372
pub const tSYMBOLS_BEG: i32 = 372
Token "symbol list", to be returned by the scanner.
Sourcepub const tQSYMBOLS_BEG: i32 = 373
pub const tQSYMBOLS_BEG: i32 = 373
Token "verbatim symbol list", to be returned by the scanner.
Sourcepub const tSTRING_END: i32 = 374
pub const tSTRING_END: i32 = 374
Token "string end", to be returned by the scanner.
Sourcepub const tSTRING_DEND: i32 = 375
pub const tSTRING_DEND: i32 = 375
Token "tRCURLY", to be returned by the scanner.
Sourcepub const tSTRING_DBEG: i32 = 376
pub const tSTRING_DBEG: i32 = 376
Token tSTRING_DBEG, to be returned by the scanner.
Sourcepub const tSTRING_DVAR: i32 = 377
pub const tSTRING_DVAR: i32 = 377
Token tSTRING_DVAR, to be returned by the scanner.
Sourcepub const tLABEL_END: i32 = 379
pub const tLABEL_END: i32 = 379
Token tLABEL_END, to be returned by the scanner.
Sourcepub const tBACK_REF2: i32 = 389
pub const tBACK_REF2: i32 = 389
Token "`", to be returned by the scanner.
Sourcepub const tUMINUS_NUM: i32 = 406
pub const tUMINUS_NUM: i32 = 406
Token tUMINUS_NUM, to be returned by the scanner.
Sourcepub const tLAST_TOKEN: i32 = 407
pub const tLAST_TOKEN: i32 = 407
Token tLAST_TOKEN, to be returned by the scanner.