Struct pm_parser

#[repr(C)]
pub struct pm_parser {

    pub node_id: u32,
    pub lex_state: pm_lex_state_t,
    pub enclosure_nesting: c_int,
    pub lambda_enclosure_nesting: c_int,
    pub brace_nesting: c_int,
    pub do_loop_stack: pm_state_stack_t,
    pub accepts_block_stack: pm_state_stack_t,
    pub lex_modes: pm_parser__bindgen_ty_1,
    pub start: *const u8,
    pub end: *const u8,
    pub previous: pm_token_t,
    pub current: pm_token_t,
    pub next_start: *const u8,
    pub heredoc_end: *const u8,
    pub comment_list: pm_list_t,
    pub magic_comment_list: pm_list_t,
    pub data_loc: pm_location_t,
    pub warning_list: pm_list_t,
    pub error_list: pm_list_t,
    pub current_scope: *mut pm_scope_t,
    pub current_context: *mut pm_context_node_t,
    pub current_hash_keys: *mut pm_static_literals_t,
    pub encoding: *const pm_encoding_t,
    pub encoding_changed_callback: pm_encoding_changed_callback_t,
    pub encoding_comment_start: *const u8,
    pub lex_callback: *mut pm_lex_callback_t,
    pub filepath: pm_string_t,
    pub constant_pool: pm_constant_pool_t,
    pub newline_list: pm_newline_list_t,
    pub integer_base: pm_node_flags_t,
    pub current_string: pm_string_t,
    pub start_line: i32,
    pub explicit_encoding: *const pm_encoding_t,
    pub current_block_exits: *mut pm_node_list_t,
    pub version: pm_options_version_t,
    pub command_line: u8,
    pub frozen_string_literal: i8,
    pub parsing_eval: bool,
    pub partial_script: bool,
    pub command_start: bool,
    pub recovering: bool,
    pub encoding_locked: bool,
    pub encoding_changed: bool,
    pub pattern_matching_newlines: bool,
    pub in_keyword_arg: bool,
    pub semantic_token_seen: bool,
    pub current_regular_expression_ascii_only: bool,
    pub warn_mismatched_indentation: bool,
}

This struct represents the overall parser. It contains a reference to the source file, as well as pointers that indicate where in the source it’s currently parsing. It also contains the most recent and current token that it’s considering.

Fields

node_id: u32

The next node identifier that will be assigned. This is a unique identifier used to track nodes such that the syntax tree can be dropped but the node can be found through another parse.

lex_state: pm_lex_state_t

The current state of the lexer.

enclosure_nesting: c_int

Tracks the current nesting of (), [], and {}.

lambda_enclosure_nesting: c_int

Used to temporarily track the nesting of enclosures to determine if a { is the beginning of a lambda following the parameters of a lambda.

brace_nesting: c_int

Used to track the nesting of braces to ensure we get the correct value when we are interpolating blocks with braces.

do_loop_stack: pm_state_stack_t

The stack used to determine if a do keyword belongs to the predicate of a while, until, or for loop.

accepts_block_stack: pm_state_stack_t

The stack used to determine if a do keyword belongs to the beginning of a block.

lex_modes: pm_parser__bindgen_ty_1

start: *const u8

The pointer to the start of the source.

end: *const u8

The pointer to the end of the source.

previous: pm_token_t

The previous token we were considering.

current: pm_token_t

The current token we’re considering.

next_start: *const u8

This is a special field set on the parser when we need the parser to jump to a specific location when lexing the next token, as opposed to just using the end of the previous token. Normally this is NULL.

heredoc_end: *const u8

This field indicates the end of a heredoc whose identifier was found on the current line. If another heredoc is found on the same line, then this will be moved forward to the end of that heredoc. If no heredocs are found on a line then this is NULL.

comment_list: pm_list_t

The list of comments that have been found while parsing.

magic_comment_list: pm_list_t

The list of magic comments that have been found while parsing.

data_loc: pm_location_t

An optional location that represents the location of the END marker and the rest of the content of the file. This content is loaded into the DATA constant when the file being parsed is the main file being executed.

warning_list: pm_list_t

The list of warnings that have been found while parsing.

error_list: pm_list_t

The list of errors that have been found while parsing.

current_scope: *mut pm_scope_t

The current local scope.

current_context: *mut pm_context_node_t

The current parsing context.

current_hash_keys: *mut pm_static_literals_t

The hash keys for the hash that is currently being parsed. This is not usually necessary because it can pass it down the various call chains, but in the event that you’re parsing a hash that is being directly pushed into another hash with **, we need to share the hash keys so that we can warn for the nested hash as well.

encoding: *const pm_encoding_t

The encoding functions for the current file is attached to the parser as it’s parsing so that it can change with a magic comment.

encoding_changed_callback: pm_encoding_changed_callback_t

When the encoding that is being used to parse the source is changed by prism, we provide the ability here to call out to a user-defined function.

encoding_comment_start: *const u8

This pointer indicates where a comment must start if it is to be considered an encoding comment.

lex_callback: *mut pm_lex_callback_t

This is an optional callback that can be attached to the parser that will be called whenever a new token is lexed by the parser.

filepath: pm_string_t

This is the path of the file being parsed. We use the filepath when constructing SourceFileNodes.

constant_pool: pm_constant_pool_t

This constant pool keeps all of the constants defined throughout the file so that we can reference them later.

newline_list: pm_newline_list_t

This is the list of newline offsets in the source file.

integer_base: pm_node_flags_t

We want to add a flag to integer nodes that indicates their base. We only want to parse these once, but we don’t have space on the token itself to communicate this information. So we store it here and pass it through when we find tokens that we need it for.

current_string: pm_string_t

This string is used to pass information from the lexer to the parser. It is particularly necessary because of escape sequences.

start_line: i32

The line number at the start of the parse. This will be used to offset the line numbers of all of the locations.

explicit_encoding: *const pm_encoding_t

When a string-like expression is being lexed, any byte or escape sequence that resolves to a value whose top bit is set (i.e., >= 0x80) will explicitly set the encoding to the same encoding as the source. Alternatively, if a unicode escape sequence is used (e.g., \u{80}) that resolves to a value whose top bit is set, then the encoding will be explicitly set to UTF-8.

The next time this happens, if the encoding that is about to become the explicitly set encoding does not match the previously set explicit encoding, a mixed encoding error will be emitted.

When the expression is finished being lexed, the explicit encoding controls the encoding of the expression. For the most part this means that the expression will either be encoded in the source encoding or UTF-8. This holds for all encodings except US-ASCII. If the source is US-ASCII and an explicit encoding was set that was not UTF-8, then the expression will be encoded as ASCII-8BIT.

Note that if the expression is a list, different elements within the same list can have different encodings, so this will get reset between each element. Furthermore all of this only applies to lists that support interpolation, because otherwise escapes that could change the encoding are ignored.

At first glance, it may make more sense for this to live on the lexer mode, but we need it here to communicate back to the parser for character literals that do not push a new lexer mode.

current_block_exits: *mut pm_node_list_t

When parsing block exits (e.g., break, next, redo), we need to validate that they are in correct contexts. For the most part we can do this by looking at our parent contexts. However, modifier while and until expressions can change that context to make block exits valid. In these cases, we need to keep track of the block exits and then validate them after the expression has been parsed.

We use a pointer here because we don’t want to keep a whole list attached since this will only be used in the context of begin/end expressions.

version: pm_options_version_t

The version of prism that we should use to parse.

command_line: u8

The command line flags given from the options.

frozen_string_literal: i8

Whether or not we have found a frozen_string_literal magic comment with a true or false value. May be:

• PM_OPTIONS_FROZEN_STRING_LITERAL_DISABLED
• PM_OPTIONS_FROZEN_STRING_LITERAL_ENABLED
• PM_OPTIONS_FROZEN_STRING_LITERAL_UNSET

parsing_eval: bool

Whether or not we are parsing an eval string. This impacts whether or not we should evaluate if block exits/yields are valid.

partial_script: bool

Whether or not we are parsing a “partial” script, which is a script that will be evaluated in the context of another script, so we should not check jumps (next/break/etc.) for validity.

command_start: bool

Whether or not we’re at the beginning of a command.

recovering: bool

Whether or not we’re currently recovering from a syntax error.

encoding_locked: bool

This is very specialized behavior for when you want to parse in a context that does not respect encoding comments. Its main use case is translating into the whitequark/parser AST which re-encodes source files in UTF-8 before they are parsed and ignores encoding comments.

encoding_changed: bool

Whether or not the encoding has been changed by a magic comment. We use this to provide a fast path for the lexer instead of going through the function pointer.

pattern_matching_newlines: bool

This flag indicates that we are currently parsing a pattern matching expression and impacts that calculation of newlines.

in_keyword_arg: bool

This flag indicates that we are currently parsing a keyword argument.

semantic_token_seen: bool

Whether or not the parser has seen a token that has semantic meaning (i.e., a token that is not a comment or whitespace).

current_regular_expression_ascii_only: bool

True if the current regular expression being lexed contains only ASCII characters.

warn_mismatched_indentation: bool

By default, Ruby always warns about mismatched indentation. This can be toggled with a magic comment.

Trait Implementations

impl Clone for pm_parser

fn clone(&self) -> pm_parser

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for pm_parser

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for pm_parser

fn default() -> Self

Returns the “default value” for a type.

impl Copy for pm_parser