Crate ezno_parser

Ezno’s Parser

Contains “string to AST” parser, AST definitions, AST back to text/string form methods and hooks for traversing/visiting AST. Used in ezno-checker and the Ezno toolchain.

use ezno_parser::{ASTNode, Expression};

fn main() {
	let expressions = [
		"4 + 2 * 5",
		"4 * 2 + 5",
		"4 * 2 * 5",
		"console.log(4 * 2, t ? true : `Hi`) == 2 && 4 == 2",
	];
	for expression in expressions {
		let expression = Expression::from_string(expression.to_owned(), Default::default());
		println!("{expression:#?}");
	}
}

Also checkout other parsers such as boa, biome, oxc and swc.

Goals

• Keep under 15k lines of code (excluding /tests and /examples folders)
• Easy to use API (see /tests and /examples folders for example usage)
• Run in WASM
• Keep readable and maintainable
• Designed for analysis and transformations
  • See expression identifiers can be used to bind information to
  • Retain source positions for use in analysis diagnostics and generating source maps
  • All AST should be visitable. Immutably to collect facts or mutable to transform/remove
• Optionally via configuration extend the ECMAScript language definition
  • TypeScript type annotations - Interfaces, enums and type alias statements - Parameter, return type and variable annotations - satisfies and as 😑
  • JSX support - Includes HTML comments, special handing of self closing tags from the specification
  • Others under feature = "extras" 👀
• Transformation and visiting
  • See example
  • ezno-parser-visitable-derive is a macro that automates/generates the visiting implementation
  • The generator macro also makes creating AST easy. See example
• Positions in source
  • All syntax has reference to where it was in the source using a Span. This uses the source-map crate crate which makes it trivial to build source maps.
• Partial AST
  • Most of the parser requires valid input. However under a option you can enable a option which can add marked nodes for cases where a expression might be missing. This allows tools that require an AST to work with a source that is still being edited (such as in a LSP).
  • It checks two cases, if either of these cases is encountered, then it adds a marker node: (1) Whether the expression ends in ) etc (useful in if etc) or (2) Whether the next token is const etc and on the next line
• Output
  • Stripping type annotations can be stripped from output using ToStringOptions { include_types: false, ..Default::default() }
  • Adding indentation under pretty: true, not adding whitespace for production builds
  • Setting max_line_length to a size to wrap certain structures
  • Support for source map mapping generation

Non-goals

• CSTs, close to source operations etc
  • Source with unbalanced parenthesis/brackets
• Increase code size or decrease readability for minor speed improvements

Testing

If in main root rather than this folder, add -p ezno-parser after cargo run to the following commands.

For testing whether the parser can lex a file

cargo run --example lex path/to/file.js

and parse

cargo run --example parse path/to/file.js

Note the Ezno CLI includes the ast-playground subcommand: a more user oriented version of these commands

Re-exports

pub use declarations::Declaration;

pub use marker::Marker;

pub use expressions::Expression;

pub use expressions::PropertyReference;

pub use functions::FunctionBase;

pub use functions::FunctionBased;

pub use functions::FunctionHeader;

pub use generator_helpers::IntoAST;

pub use property_key::PropertyKey;

pub use statements::Statement;

pub use types::type_annotations;

pub use types::type_annotations::TypeAnnotation;

pub use types::type_declarations;

pub use types::type_declarations::TypeParameter;

pub use source_map;

pub use options::*;

Modules

ast

Re-exports or generator and general use

declarations

expressions

functions

generator_helpers

is_expression

marker

number

options

property_key

statements

types

Includes type annotations + syntax added by TypeScript (and Ezno) such as declare declarations

visiting

Contains logic that makes viewing and transform parts of AST trivial through the visitor pattern

Structs

Block

A “block” of braced statements and declarations

BlockLike

BlockLikeMut

Decorated

TODO under cfg if don’t want this could just be type Decorated<T> = T;

Decorator

ExpressionPosition

JSXElement

JSXFragment

KeywordPositions

As parsing is forwards, this is ordered

LexerOptions

LocalToStringInformation

Module

ParseError

A error for not parsing

ParsingState

SourceId

A identifier for a crate::Source

SpreadDestructuringField

Covers [ArrayDestructuring] AND ObjectDestructuringField

StatementPosition

Enums

ArrayDestructuringField

For

BlockOrSingleStatement

For ifs and other statements

JSXAttribute

TODO spread attributes and boolean attributes

JSXElementChildren

JSXNode

JSXRoot

ObjectDestructuringField

For

ParseErrors

Quoted

What surrounds a string

StatementOrDeclaration

TSXKeyword

TSXToken

All JS Tokens with extensions including TypeScript, JSX and more

VariableField

A variable declaration name, used in variable declarations and function parameters. See destructuring

VariableIdentifier

VariableKeyword

WithComment

Traits

ASTNode

Defines common methods that would exist on a AST part include position in source, creation from reader and serializing to string from options.

DestructuringFieldInto

ExpressionOrStatementPosition

Classes and function functions have two variants depending whether in statement position or expression position

ListItem

Functions

are_nodes_over_length

TODO WIP!

html_tag_contains_literal_content

Used for lexing

html_tag_is_self_closing

Used for lexing

script_to_tokens

For demos and testing

Type Aliases

ParseResult

Span