Jsompiler Lexer

The jsompiler_lexer is responsible for the lexical analysis part of the jsompiler project. It takes raw JavaScript source code and converts it into a stream of tokens, which are used by the parser for syntax analysis. Note: This project is currently under active development. Interfaces and behavior may change as the project evolves.

Features

Tokenization of keywords, identifiers, numbers, and strings
Support for almost all JavaScript operators and delimiters
Error handling for invalid lexemes
Easily extendable token definitions
Limitation: Unicode escape sequences (e.g., \u0041) are not yet supported.

Installation

Add jsompiler_lexer to your Cargo.toml:

[dependencies]
jsompiler_lexer = "0.2.1"

OR Run the following Cargo command in your project directory:

cargo add jsompiler_lexer

Example

const pi = 3.1416;

Lexemes	Tokens
const	Keyword(Const)
pi	Identifier("pi")
=	Operator(EqualTo)
3.1416	Literal(Number(Value(3.1416)))
;	Delimiter(Semicolon)
EOF	EOF

Usage

Add this crate to your project

use jsompiler_lexer::Lexer;

let source = "const pi = 3.1416;";
let mut lexer = Lexer::new(source);
lexer.scan_all_tokens();
println!("{:#?}", lexer.tokens);
println!("{:#?}", lexer.errors);

API Documentation

Tokens Supported

Keywords:

const, function, void, typeof, if, return, else, instanceof, in, while, for, do, true, false, exports, break, continue, switch, null, case, debugger, class, new, this, super, import, export, default, try, catch, finally, throw, enum, extends, delete

Contextual Keywords:

let, package, interface, get, set, arguments, public, eval, protected, private, meta, target, async, await, from, as, from, as, of, yield, static, with, implements

Operators

Arithmetic Operators

+, -, *, /, %, **, ++, --

Assignment Operators

=, +=, -=, *=, /=, %=, **=, &=, |=, ^=, <<=, >>=, >>>=, &&=, ||=, ??=

Comparison Operators

==, ===, !=, !==, >, <, >=, <=

Logical Operators

&&, ||, !, ??

Bitwise Operators

&, |, ^, ~, <<, >>, >>>

Miscellaneous Operators

=>, ?, :, ?., ...

Delimiters

(, ), {, } ,[, ], ;, ,, .

Comments

//, /*

Literal Types

undefined, null, NaN, Infinity

jsompiler_lexer 0.2.2