Crate lib_ruby_parser[−][src]
Expand description
lib-ruby-parser
lib-ruby-parser
is a Ruby parser written in Rust.
Basic usage:
use lib_ruby_parser::{Parser, ParserOptions};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let options = ParserOptions {
buffer_name: "(eval)".to_string(),
..Default::default()
};
let mut parser = Parser::new(b"2 + 2".to_vec(), options);
println!("{:#?}", parser.do_parse());
Ok(())
}
Features
TLDR; it’s fast, it’s precise, and it has a beautiful interface.
Comparison with Ripper
/RubyVM::AST
:
- It’s based on MRI’s
parse.y
, and so it returns exactly the same sequence of tokens. - It’s been tested on top 300 gems (by total downlads, that’s about 3M LOC),
rubyspec
andruby/ruby
repos and there’s no difference withRipper.lex
. - It’s ~3 times faster than
Ripper
(withjemalloc
), Ripper parses 3.9M LOC in ~16s,lib-ruby-parser
does it in ~6.5s. That’s ~600K LOC/s. You can find some benchmarks in thebench/
directory, they don’t include IO and GC. - It has a much, much better interface. AST is strongly typed and well documented.
- It doesn’t throw away information about tokens. All nodes have information about their source locations.
Comparison with whitequark/parser:
- It’s much faster (the same corpus of 3M LOC can be parsed in 180s on the same machine)
- It has a very similar interface (both in terms of AST structure and errors reporting)
- However, AST is strongly typed, and so if something is nullable it’s explicitly defined and documented.
- What’s important, it doesn’t depend on Ruby
Grammar versioning
lib-ruby-parser
follows MRI/master. There are no plans to support multiple versions like it’s done in whitequark/parser
.
Library versioning
Ruby version | lib-ruby-parser version |
---|---|
3.0.0 | 3.0.0+ |
Encodings
By default lib-ruby-parser
can only parse source files encoded in UTF-8
or ASCII-8BIT/BINARY
.
It’s possible to pass a decoder
function in ParserOptions
that takes a recognized (by the library) encoding and a byte array. It must return a UTF-8 encoded byte array or an error:
use lib_ruby_parser::source::{InputError, Decoder, DecoderResult};
use lib_ruby_parser::{Parser, ParserOptions, ParserResult};
fn decode(encoding: String, input: Vec<u8>) -> DecoderResult {
if "US-ASCII" == encoding.to_uppercase() {
// reencode and return Ok(result)
return DecoderResult::Ok(b"# encoding: us-ascii\ndecoded".to_vec());
}
DecoderResult::Err(InputError::DecodingError(
"only us-ascii is supported".to_string(),
))
}
let options = ParserOptions {
decoder: Some(Decoder::new(Box::new(decode))),
..Default::default()
};
let mut parser = Parser::new(b"# encoding: us-ascii\n3 + 3".to_vec(), options);
let ParserResult { ast, input, .. } = parser.do_parse();
assert_eq!(ast.unwrap().expression().source(&input).unwrap(), "decoded".to_string())
Invalid string values
Ruby doesn’t require string literals to be valid in their encodings. This is why the following code is valid:
"\xFF"
Byte sequence 255
is invalid in UTF-8, but MRI ignores it.
But not all languages support it, and this is why string and symbol nodes encapsulate a custom StringValue
instead of a plain String
.
If your language supports invalid strings you can use raw .bytes
of this StringValue
. For example, a Ruby wrapper for this library could do that.
If your language doesn’t support it, better call .to_string_lossy()
that replaces all unsupported chars with a special U+FFFD REPLACEMENT CHARACTER (�)
.
Regexes
Ruby constructs regexes from literals during parsing to:
- validate them
- declare local variables if regex is used for matching AND it contains named captures
To mirror this behavior lib-ruby-parser
uses Onigurama to compile, validate and parse regex literals.
This feature is disabled by default, but you can add it by enabling "onig"
feature.
Bison
The grammar of lib-ruby-parser
is built using a custom bison skeleton that was written for this project.
For development you need the latest version of Bison installed locally. Of course, it’s not necessary for release builds from crates.io (because compiled parser.rs
is included into release build AND build.rs
that converts it is excluded).
If you use it from GitHub directly you also need Bison (because parser.rs
is under gitignore)
Bindings for other languages
Profiling
You can use parse
example:
$ cargo run --features run-examples --example parse -- --print=N --run-profiler=1 "<pattern>"
Benchmarking
A codebase of 3.9M LOCs can be generated using a download.rb
script:
$ ruby gems/download.rb
Then, run a script that compares Ripper
and lib-ruby-parser
(attached results are from Feb 2021):
$ ./bench/compare.sh
Finished release [optimized] target(s) in 0.08s
Running lib-ruby-parser
Run 1:
Time taken: 6.6232788220 (total files: 18018)
Run 2:
Time taken: 6.6498335800 (total files: 18018)
Run 3:
Time taken: 7.0684415810 (total files: 18018)
Run 4:
Time taken: 6.7987308510 (total files: 18018)
Run 5:
Time taken: 6.6954798760 (total files: 18018)
--------
Running MRI/ripper
Run 1:
Time taken: 22.92822499992326 (total files: 18017)
Run 2:
Time taken: 21.8613000002224 (total files: 18017)
Run 3:
Time taken: 21.96083900006488 (total files: 18017)
Run 4:
Time taken: 21.44488099985756 (total files: 18017)
Run 5:
Time taken: 21.738944000098854 (total files: 18017)
Profile-guided optimization
RUSTFLAGS="-Cprofile-generate=$(PWD)/target/pgo/pgo.profraw" cargo build --release --example parse
target/release/examples/parse --no-output "gems/repos/**/*.rb"
llvm-profdata merge -o target/pgo/pgo.profraw/merged.profdata target/pgo/pgo.profraw
RUSTFLAGS="-Cprofile-use=$(PWD)/target/pgo/pgo.profraw/merged.profdata" cargo build --release --example parse
PGO, No LTO:
$ repeat 5 time target/release/examples/parse --no-output "gems/repos/**/*.rb"
9.46s user 1.27s system 80% cpu 13.371 total
8.51s user 0.66s system 99% cpu 9.171 total
8.52s user 0.68s system 99% cpu 9.208 total
9.63s user 0.74s system 99% cpu 10.381 total
9.70s user 0.73s system 99% cpu 10.443 total
No PGO, LTO=fat:
$ repeat 5 time target/release/examples/parse --no-output "gems/repos/**/*.rb"
9.90s user 1.29s system 80% cpu 13.917 total
9.42s user 0.71s system 99% cpu 10.138 total
10.24s user 0.76s system 99% cpu 11.004 total
10.21s user 0.75s system 99% cpu 10.962 total
10.22s user 0.74s system 99% cpu 10.966 total
The diff seems to be too small to use this feature.
When both PGO and LTO are enabled building a parse
example gives a bunch of LLVM errors about wrong types of functions (like expected a Function or null
).
If you know how to fix them, please open an issue.
Fuzz testing
First, make sure to switch to nightly:
$ rustup default nightly
Then install cargo-fuzz
:
$ cargo install cargo-fuzz
And run the fuzzer (change the number of --jobs
as you need or remove it to run only 1 parallel process):
$ RUST_BACKTRACE=1 cargo fuzz run parse --jobs=8 -- -max_len=50
A Ruby parser written in Rust.
Uses bison under the hood.
Modules
Module with generic containers
Mod with all known lex states
Module with all known node types
Module with everything related to output of the Parser, but not related to AST,
like Comment
, Input
, Decoder
Module to perform recursive traversing
Structs
Representation of a byte sequence
Diagnostic message that comes from the parser when there’s an error or warning
State of the lexer
A struct responsible for converting a given input into a sequence of tokens
Representation of any location in the given input
A Bison parser, automatically generated from src/parser/parser.y.
Configuration of the parser
Combination of all data that Parser
can give you
Representation of a reserved word in Ruby
Stack of local variables in nested scopes
A token that is emitted by a lexer and consumed by a parser
Enums
Enum of all possible diagnostic message (both warnings and errors)
Error level of the diagnostic message
Generic combination of all known nodes.
Functions
Returns a ReservedWord
for a given string slice.