Module parser

Parser

If the tokenizer is responsible for dividing large pieces of text into smaller chunks, then the parser is responsible for rebuilding and reorganizing those chunks into larger trees. Parser definitions are specified through a syntax that combines smaller patterns into larger ones, gradually building on each other to form a parse tree.

This parse tree is returned as a ParseNode, which acts as a tree. Each node contains a range of indices and a token corpus (representing the text the parse node corresponds to), along with a list of children.

Syntax Reference

As with the tokenizer, parser definitions are stored in the TOML format as string values. (Note that these are stored under the [parser] header, not the [tokenizer] one.) The following syntax elements can be used in parser definitions:

[parser]

name_reference = "token_name"   # Include the name of a token or other parse
                                # rule to reference it
optional = "foo?"
zero_or_more = "foo*"
one_or_more = "foo+"
multiple = "(foo bar baz)"      # Checks for multiple patterns in sequence
choose = "[foo bar baz]"        # Checks for any of the patterns

These basic patterns can be combined for more complicated rules.

[parser]

complex_pattern = "(foo* [bar baz]? qux)+"

Once a parse tree is obtained, the next step is evaluating the tree to extract semantic information. This is usually done by recursively querying the contents and children of parse nodes through functions like ParseNode::find (to find the descendant of a node through a dot-separated path).

Modules

meta

This module contains the “meta-parser”, which parses human-readable parse instructions from TOML tables and converts them to instructions for the Parser.

rule

This module defines the parser rules, which determine how tokens are transformed in the crate::parser.

Structs

ParseDefinition

An entry in a parser that defines one type or pattern of node.

ParseNode

A parse tree, containing its children and the text it represents.

ParseNodeChild

Represents a ParseNode’s child, containing a name (to be used in methods like ParseNode::find) and another node.

Parser

Represents a list of parser definitions along with any necessary settings.