RESS
Rusty EcmaScript Scanner
A scanner/tokenizer for JS written in Rust
Usage
There are two main interfaces for using ress
in your Rust code.
The first is the very simple function tokenize
, this takes in a String
and outputs a Vec<Token>
.
extern crate ress;
use tokenize;
static JS: &str = include_str!;
The other option is to create a Scanner
, an iterator over the Item
struct. Item
has two fields token
for the RefToken
found and Span
for the position in the string.
extern crate ress;
use ;
static JS: &str = include_str!;
In either method the major construct that you would be dealing with is a RefToken
enum. This enum represents the 10 different tokens defined in the ECMAScript specification.
ES Tokens
- Boolean Literal
- End of File
- Identifier
- Keyword
- Null Literal
- Numeric Literal
- Punctuation
- String Literal
- Regular Expression Literal
- Comment
Keep in mind that keywords have been moving around a lot in JS between ES3 through ES2019 so you might find some items parsed as keywords in the ES2019 context that are not in the ES3 context and since my goal is keep this scanner context free this should be dealt with at a higher level. A good example of this is yield
which is sometimes a keyword and sometimes an identifier, this package will always parse this as a Keyword. Please also note that any pending specifications at tc39 may not be included
For each of the token cases there is either a struct or enum to provide additional information with the exception of NullLiteral
and EoF
which should be self explanatory. The more complicated items do implement ToString
which should get you back to the original js text for that token. The Token
enum also provides a number of helper functions for building that picture without pulling the inner data our of the enum. Using the Punct
case as an example the helper functions look like this
;
;
;
A similar set of functions are available for each case. Be aware that some _str
implementations panic if the wrong string is provided meaning these would also panic.
let p = Punct;
if p.matches_keyword_str
if p.matches_keyword
Like all Iterators
the Scanner
has a next
method, It also has a look_ahead
method that will allow you to parse the next value without advancing. Using this method can be a convenient way to get the next token without performing a mutable borrow, however you will be incurring the cost of parsing that token twice. All Iterators
implement Peekable
that will convert them into a new iterator with a peek
method, this will allow you to look ahead while only paying the cost once however peek
performs a mutable borrow which means it needs to be in a different scope than a call to next
.
// look_ahead
let js = "function() { return; }";
let mut s = new;
let current = s.next;
let next = s.look_ahead;
let new_current = s.next;
assert_eq!;
// peekable (fails to compile)
let p = new.peekable;
let current = s.next; // <-- first mutable borrow
let next = p.peek; // <-- second mutable borrow
For more intense lookahead scenarios Scanner
makes available the get_state
and set_state
methods. These methods will allow you to capture a snapshot of the current position and any context, and then later reset to that position and context.
let js = "function() {
return 0;
};";
let mut s = new;
let start = s.get_state;
assert_eq!;
assert_eq!;
assert_eq!;
s.set_state;
assert_eq!;
In addition to the RefToken
enum this crate also provides an OwnedToken
enum, this will semantically work the same as a RefToken
(as they both impl Token
) however it will require a string be allocated to construct. This can be useful if you need to work outside of the lifetime bounds of a RefToken
.
Why?
Wouldn't it be nice to write new JS development tools in Rust? The (clear-comments)[https://github.com/FreeMasen/RESS/blob/master/examples/clear-comments/src/main.rs] example is a proof of concept on how you might use this crate to do just that. This example will take in a JS file and output a version with all of the comments removed. An example of how you might see it in action is below (assuming you have a file called in.js in the project root).
Performance
The below stats are from running cargo +nightly bench
on a MBP (2.9 GHz i9-8850H & 16bg RAM).
Lib | Size | Time | +/- |
---|---|---|---|
Angular 1.5 | 1.16mb | 18.991 ms | 4.393 ms |
jquery | 271.75kb | 7.218 ms | 577.236 μs |
React | 59.09kb | 1.976 ms | 116.139 μs |
React-dom | 641.51kb | 16.880 ms | 3.614 ms |
Vue | 289.30kb | 9.675 ms | 1.402 ms |
If you are interested in getting an idea about performance without waiting for cargo bench
to complete you can run the following command.