Crate rslint_parser

Source
Expand description

Extremely fast, lossless, and error tolerant JavaScript Parser.

The parser uses an abstraction over non-whitespace tokens. This allows us to losslessly or lossly parse code without requiring explicit handling of whitespace. The parser yields events, not an AST, the events are resolved into untyped syntax nodes, which can then be casted into a typed AST.

The parser is able to produce a valid AST from any source code. Erroneous productions are wrapped into ERROR syntax nodes, the original source code is completely represented in the final syntax nodes.

You probably do not want to use the parser struct, unless you want to parse fragments of Js source code or make your own productions. Instead use functions such as parse_text and parse_text_lossy which offer abstracted versions for parsing.

Notable features of the parser are:

  • Extremely fast parsing and lexing through the extremely fast rslint_lexer.
  • Ability to do Lossy or Lossless parsing on demand without explicit whitespace handling.
  • Customizable, able to parse any fragments of JS code at your discretion.
  • Completely error tolerant, able to produce an AST from any source code.
  • Zero cost for converting untyped nodes to a typed AST.
  • Ability to go from AST to SyntaxNodes to SyntaxTokens to source code and back very easily with nearly zero cost.
  • Very easy tree traversal through SyntaxNode.
  • Descriptive errors with multiple labels and notes.
  • Very cheap cloning, cloning an ast node or syntax node is the cost of adding a reference to an Rc.
  • Cheap incremental reparsing of changed text.

The crate further includes utilities such as:

It is inspired by the rust analyzer parser but adapted for JavaScript.

§Syntax Nodes vs AST Nodes

The crate relies on a concept of untyped SyntaxNodes vs typed AstNodes. Syntax nodes represent the syntax tree in an untyped way. They represent a location in an immutable tree with two pointers. The syntax tree is composed of SyntaxNodes and SyntaxTokens in a nested tree structure. Each node can have parents, siblings, children, descendants, etc.

AstNodes represent a typed version of a syntax node. They have the same exact representation as syntax nodes therefore a conversion between either has zero runtime cost. Every piece of data of an ast node is optional, this is due to the fact that the parser is completely error tolerant.

Each representation has its advantages:

§SyntaxNodes

  • Very simple traversing of the syntax tree through functions on them.
  • Easily able to convert to underlying text, range, or tokens.
  • Contain all whitespace bound to the underlying production (in the case of lossless parsing).
  • Can be easily converted into its typed representation with zero cost.
  • Can be turned into a pretty representation with fmt debug.

§AST Nodes

  • Easy access to properties of the underlying production.
  • Zero cost conversion to a syntax node.

In conclusion, the use of both representations means we are not constrained to acting through typed nodes. Which makes traversal hard and you often have to resort to autogenerated visitor patterns. AST nodes are simply a way to easily access subproperties of a syntax node.event;

Re-exports§

pub use crate::ast::AstNode;
pub use crate::ast::AstToken;
pub use crate::util::SyntaxNodeExt;
pub use crate::util::SyntaxTokenExt;

Modules§

ast
AST definitions for converting untyped syntax nodes into typed AST nodes.
syntax
The Js syntax itself and parser functions.
util
Extra utlities for untyped syntax nodes, syntax tokens, and AST nodes.

Macros§

T
Utility macro for creating a SyntaxKind through simple macro syntax
at_ident_name
match_ast
Matches a SyntaxNode against an ast type.
op
A simple macro for making assign, binop, or unary operators
token_set
Utility macro for making a new token set

Structs§

BigInt
A big signed integer type.
Checkpoint
A structure signifying the Parser progress at one point in time
CompletedMarker
A structure signifying a completed node
GreenNode
Internal node in the immutable tree. It has other nodes and tokens as children.
JsLanguage
LosslessTreeSink
Structure for converting events to a syntax tree representation, while preserving whitespace.
LossyTreeSink
Structure to convert events to a lossy syntax tree which does not preserve whitespace.
Marker
A structure signifying the start of parsing of a syntax tree node
Parse
A utility struct for managing the result of a parser job
Parser
An extremely fast, error tolerant, completely lossless JavaScript parser
ParserState
State kept by the parser while parsing. It is required for things such as strict mode or async functions
SmolStr
A SmolStr is a string type that has the following properties:
Syntax
A structure describing the syntax features the parser will accept. The default is an ECMAScript 2021 Script without any proposals.
SyntaxText
SyntaxTreeBuilder
Simple wrapper around a rslint_rowan GreenNodeBuilder
TextRange
A range in text, represented as a pair of TextSize.
TextSize
A measure of text length. Also, equivalently, an index into text.
Token
Abstracted token for TokenSource
TokenSet
TokenSource
The source of tokens for the parser

Enums§

Direction
Event
Events emitted by the Parser, these events are later made into a syntax tree with process into TreeSink.
FileKind
The kind of file we are parsing
JsNum
NodeOrToken
StrictMode
SyntaxKind
The kind of syntax node, e.g. IDENT, FUNCTION_KW, or FOR_STMT.
WalkEvent
WalkEvent describes tree walking process.

Traits§

TreeSink
An abstraction for syntax tree implementations

Functions§

parse_expr
Losslessly Parse text into an expression Parse which can then be turned into an untyped root SyntaxNode. Or turned into a typed Expr with tree.
parse_js_num
Parse a js number as a string into a number.
parse_module
Same as parse_text but configures the parser to parse an ECMAScript module instead of a script
parse_module_lossy
Same as parse_text_lossy but configures the parser to parse an ECMAScript module instead of a Script
parse_text
Parse text into a Parse which can then be turned into an untyped root SyntaxNode. Or turned into a typed Script with tree.
parse_text_lossy
Lossly parse text into a Parse which can then be turned into an untyped root SyntaxNode. Or turned into a typed Script with tree.
parse_with_syntax
process
Generate the syntax tree with the control of events.
tokenize
Run the rslint_lexer lexer to turn source code into tokens and errors produced by the lexer

Type Aliases§

ParserError
The type of error emitted by the parser, this includes warnings, notes, and errors.
It also includes labels and possibly notes
SyntaxElement
SyntaxElementChildren
SyntaxNode
SyntaxNodeChildren
SyntaxToken