syn-grammar

syn-grammar is a powerful parser generator for Rust that allows you to define EBNF-like grammars directly inside your code. It compiles these definitions into efficient syn parsers at compile time.

Writing parsers for procedural macros or Domain Specific Languages (DSLs) in Rust often involves writing repetitive boilerplate code using the syn crate. syn-grammar simplifies this process by letting you describe what you want to parse using a clean, readable syntax, while handling the complex logic of parsing, backtracking, and error reporting for you.

Features

• Inline Grammars: Define your grammar directly in your Rust code using the grammar! macro.
• EBNF Syntax: Familiar syntax with sequences, alternatives (|), optionals (?), repetitions (*, +), and grouping (...).
• Type-Safe Actions: Directly map parsing rules to Rust types and AST nodes using action blocks (-> { ... }).
• Seamless Syn Integration: First-class support for parsing Rust tokens like identifiers, literals, types, and blocks.
• Automatic Left Recursion: Write natural expression grammars (e.g., expr = expr + term) without worrying about infinite recursion.
• Backtracking & Ambiguity: Automatically handles ambiguous grammars with speculative parsing.
• Cut Operator: Control backtracking explicitly for better error messages and performance.
• Rule Arguments: Pass context or parameters between rules.
• Grammar Inheritance: Reuse rules from other grammars.
• Testing Utilities: Fluent API for testing your parsers.

Installation

1. Quick Installation (Runtime Parsing)

Use this setup if you want to parse strings at runtime inside your application. This is the standard approach for:

• CLIs & Interpreters: Parsing user input or commands.
• Configuration Files: Reading custom config formats at startup.
• Prototyping: Experimenting with grammars in main.rs.

Add syn-grammar and syn to your Cargo.toml. syn is required at runtime because the generated parser relies on its types (e.g., ParseStream, Ident).

[dependencies]
syn-grammar = "0.3"
syn = { version = "2.0", features = ["full", "extra-traits"] }
quote = "1.0"
proc-macro2 = "1.0"

2. Optimized Installation (Compile-Time Macros)

If you are writing a procedural macro to parse input at compile time, you should isolate your parser definition in a separate crate. This is the correct approach for:

• Embedded DSLs: Parsing custom syntax inside Rust code (e.g., HTML-like templates, State Machines, SQL-like queries).
• Code Generation: Reading an external definition file during the build and generating Rust code from it.
• Compile-Time Verification: Checking syntax or configuration validity during cargo build.

Steps:

1. Create a separate proc-macro crate.
2. Add syn-grammar, syn, and quote to that crate's Cargo.toml.
3. Define your grammar and macro there.
4. Depend on that crate from your main project.

Why? Your main project will use the macro to generate code, but the heavy syn parsing logic will not be compiled into your final binary. This significantly improves build times for users of your macro.

⚠️ Important Note on Tokenization

Since syn-grammar is built on top of syn, it uses the Rust Tokenizer. This means your grammar must consist of valid Rust tokens.

• Good Use Cases: Grammars that look somewhat like code or data structures (e.g., JSON, mathematical expressions, C-like syntax, HTML tags).
• Limitations: You cannot parse languages that require a custom lexer, such as:
  • Whitespace-sensitive languages (e.g., Python, YAML) — syn skips whitespace automatically.
  • Binary formats.
  • Arbitrary text that doesn't form valid Rust tokens (e.g., unquoted strings with special characters like @ or $ in positions Rust doesn't allow).

Quick Start

Here is a complete example of a calculator grammar that parses mathematical expressions into an i32.

use syn_grammar::grammar;
use syn::parse::Parser; // Required for .parse_str()

grammar! {
    grammar Calc {
        // The return type of the rule is defined after `->`
        pub rule expression -> i32 =
            l:expression "+" r:term -> { l + r }
          | l:expression "-" r:term -> { l - r }
          | t:term                  -> { t }

        rule term -> i32 =
            f:factor "*" t:term -> { f * t }
          | f:factor "/" t:term -> { f / t }
          | f:factor            -> { f }

        rule factor -> i32 =
            i:integer           -> { i }
          | paren(e:expression) -> { e }
    }
}

fn main() {
    // The macro generates a module `Calc` containing a function `parse_expression`
    // corresponding to the `expression` rule.
    let result = Calc::parse_expression.parse_str("10 - 2 * 3");
    assert_eq!(result.unwrap(), 4);
}

What happens under the hood?

The grammar! macro expands into a Rust module (named Calc in the example) containing:

• A function parse_<rule_name> for each rule (e.g., parse_expression).
• These functions take a syn::parse::ParseStream and return a syn::Result<T>.
• All necessary imports and helper functions to make the parser work.

Detailed Syntax Guide

Rules

A grammar consists of a set of rules. Each rule has a name, a return type, and a pattern to match.

rule name -> ReturnType = pattern -> { action_code }

• name: The name of the rule (e.g., expr).
• ReturnType: The Rust type returned by the rule (e.g., Expr, i32, Vec<String>).
• pattern: The EBNF pattern defining what to parse.
• action_code: A Rust block that constructs the return value from the bound variables.

Attributes and Doc Comments

Rules can be decorated with standard Rust attributes and documentation comments. These are passed through to the generated function.

/// Parses a valid identifier.
#[cfg(feature = "extra")]
rule ident -> Ident = ...

Rule Arguments

Rules can accept arguments, allowing you to pass context or state down the parser chain.

rule main -> i32 = 
    "start" v:value(10) -> { v }

rule value(offset: i32) -> i32 =
    i:integer -> { i + offset }

Grammar Inheritance

You can inherit rules from another grammar module. This is useful for splitting large grammars or reusing common rules.

// In base.rs
grammar! {
    grammar Base {
        pub rule num -> i32 = i:integer -> { i }
    }
}

// In derived.rs
use crate::base::Base;

grammar! {
    grammar Derived : Base {
        rule main -> i32 = 
            "add" a:num b:num -> { a + b }
    }
}

Patterns

Literals and Keywords

Match specific tokens using string literals.

rule kw -> () = "fn" "name" -> { () }

Built-in Parsers

syn-grammar provides several built-in parsers for common Rust tokens:

Sequences and Bindings

Match a sequence of patterns. Use name:pattern to bind the result to a variable available in the action block.

rule assignment -> Stmt = 
    name:ident "=" val:expr -> { 
        Stmt::Assign(name, val) 
    }

Alternatives (|)

Match one of several alternatives. The first one that matches wins.

rule boolean -> bool = 
    "true"  -> { true }
  | "false" -> { false }

Repetitions (*, +, ?)

• pattern*: Match zero or more times. Returns a Vec.
• pattern+: Match one or more times. Returns a Vec.
• pattern?: Match zero or one time. Returns an Option (or () if unbound).

rule list -> Vec<i32> = 
    "[" elements:integer* "]" -> { elements }

Groups (...)

Group patterns together to apply repetitions or ensure precedence.

rule complex -> () = 
    ("a" | "b")+ "c" -> { () }

Delimiters

Match content inside delimiters.

• paren(pattern): Matches ( pattern ).
• bracketed[pattern]: Matches [ pattern ].
• braced{pattern}: Matches { pattern }.

rule tuple -> (i32, i32) = 
    paren(a:integer "," b:integer) -> { (a, b) }

Error Recovery (recover)

You can make your parser robust against errors using recover(rule, sync_token). If rule fails, the parser will skip tokens until it finds sync_token, returning None (or (None, ...) for bindings). Note that recover does not consume the sync token.

rule stmt -> Option<Stmt> =
    // If `parse_stmt` fails, skip until `;`
    // `s` will be `Option<Stmt>` (Some if success, None if recovered)
    s:recover(parse_stmt, ";") ";" -> { s }

The Cut Operator (=>)

The cut operator => allows you to commit to a specific alternative. If the pattern before the => matches, the parser will not backtrack to try other alternatives, even if the pattern after the => fails. This produces better error messages.

rule stmt -> Stmt =
    // If we see "let", we commit to this rule. 
    // If "mut" or the identifier is missing, we error immediately 
    // instead of trying the next alternative.
    "let" => "mut"? name:ident "=" e:expr -> { ... }
  | e:expr -> { ... }

Testing

syn-grammar provides a fluent testing API via the grammar-kit crate (re-exported as syn_grammar::testing).

use syn_grammar::testing::Testable;

#[test]
fn test_calc() {
    Calc::parse_expression
        .parse_str("1 + 2")
        .test()
        .assert_success_is(3);

    Calc::parse_expression
        .parse_str("1 + *")
        .test()
        .assert_failure_contains("expected term");
}

Advanced Topics

Left Recursion

Recursive descent parsers typically struggle with left recursion (e.g., A -> A b). syn-grammar automatically detects direct left recursion and compiles it into an iterative loop. This makes writing expression parsers natural and straightforward.

// This works perfectly!
rule expr -> i32 = 
    l:expr "+" r:term -> { l + r }
  | t:term            -> { t }

Backtracking

By default, syn-grammar uses syn's speculative parsing (fork) to try alternatives.

1. It checks if the next token matches the start of an alternative (using peek).
2. If ambiguous, it attempts to parse the alternative.
3. If it fails, it backtracks and tries the next one.

This allows for flexible grammars but can impact performance if overused. Use the Cut Operator (=>) to prune the search space when possible.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.