fmt-lang 0.2.0

The model

• Rules describe a language's layout as data: NodeRules (group, indent, spacing before and after a node, list delimiters and separators with a trailing-separator policy, blank-line caps) and TokenRules (spacing before and after a token kind, everywhere or inside one node kind).
• Rules::compile resolves the kind names against a language once and returns a Style.
• format formats a tree with a style at a width. format_keeping also takes the regions of lexical errors to leave as written; format_doc returns the pretty_lang::Doc instead of a string.

What it guarantees, and how each guarantee is checked:

The property tests run 400 cases each by default; before this release they were soaked at 400,000 cases each.

Installation

[dependencies]
fmt-lang = "0.2"

Without the standard library:

[dependencies]
fmt-lang = { version = "0.2", default-features = false }

Quick start

A JSON-like language forged with lang-forge, and its whole formatter:

use fmt_lang::{format, Indent, NodeRule, Rules, Space, TokenRule, Trailing};
use lang_forge::Language;

let json = Language::from_lsf(r#"
    [language]
    name = "json"
    [lexer]
    strings = ['"']
    line_comments = ["//"]
    [rules]
    document = "value"
    value    = "object | array | STRING | NUMBER | 'true' | 'false' | 'null'"
    object   = "'{' (member (',' member)* ','?)? '}'"
    member   = "STRING ':' value"
    array    = "'[' (value (',' value)* ','?)? ']'"
"#)?;

let style = Rules::new()
    .indent(2)
    .verbatim("ERROR")
    .token(TokenRule::new(":").before(Space::None).after(Space::Single))
    .node(NodeRule::new("object").group().indent(Indent::Block)
        .delimiters("{", "}", Space::Line)
        .separator(",", Space::Line, Trailing::Never))
    .node(NodeRule::new("array").group().indent(Indent::Block)
        .delimiters("[", "]", Space::SoftLine)
        .separator(",", Space::Line, Trailing::Never))
    .compile(|name| json.kind(name))?;

let parse = json.parse(r#"{"tags":["a","b",],  "ok" :true // done
}"#);

assert_eq!(
    format(parse.tree(), parse.source(), &style, 80)?,
    "{\n  \"tags\": [\"a\", \"b\"],\n  \"ok\": true // done\n}\n",
);
# Ok::<(), Box<dyn std::error::Error>>(())

The line comment must end its line, so the object cannot be laid out flat and breaks, one member per line; the array still fits and stays flat. Without the comment the whole object fits on one line at width 80; at width 12 the array breaks too.

Comments stay where they belong

Comments are trailing (on the line of the token before them), leading (on their own line before the next token), or dangling (on their own line at the end of a block, indented with it):

use fmt_lang::{format, Indent, NodeRule, Rules, Space, TokenRule, Trailing};
use lang_forge::Language;

let json = Language::from_lsf(r#"
    [language]
    name = "json"
    [lexer]
    strings = ['"']
    line_comments = ["//"]
    block_comments = [["/*", "*/"]]
    [rules]
    document = "value"
    value    = "object | array | STRING | NUMBER | 'true' | 'false' | 'null'"
    object   = "'{' (member (',' member)* ','?)? '}'"
    member   = "STRING ':' value"
    array    = "'[' (value (',' value)* ','?)? ']'"
"#)?;
let style = Rules::new()
    .indent(2)
    .token(TokenRule::new(":").before(Space::None).after(Space::Single))
    .node(NodeRule::new("object").group().indent(Indent::Block)
        .delimiters("{", "}", Space::Line)
        .separator(",", Space::Line, Trailing::Preserve))
    .compile(|name| json.kind(name))?;

let src = "{\n// leading\n\"a\":1, /* trailing */\n\n\n\"b\":2\n    // dangling\n}";
let parse = json.parse(src);
assert_eq!(
    format(parse.tree(), parse.source(), &style, 80)?,
    "{\n  // leading\n  \"a\": 1, /* trailing */\n\n  \"b\": 2\n  // dangling\n}\n",
);
# Ok::<(), Box<dyn std::error::Error>>(())

Expression languages

Token rules give operators their spacing, a node rule makes a prefix operator hug its operand, and a hanging indent wraps long expressions. Nothing ever fuses two tokens: - -3 keeps its space.

use fmt_lang::{format, Indent, NodeRule, Rules, Space, TokenRule};
use lang_forge::Language;

let calc = Language::from_lsf(r#"
    [language]
    name = "calc"
    [rules]
    program = "stmt*"
    stmt    = "'let' IDENT '=' expr ';' | expr ';'"
    group   = "'(' expr ')'"
    [rules.expr]
    operand = "NUMBER | IDENT | group"
    levels  = [ { left = ["+", "-"] }, { left = ["*", "/"] }, { prefix = ["-"] } ]
"#)?;

let mut rules = Rules::new()
    .token(TokenRule::new("let").after(Space::Single))
    .token(TokenRule::new("=").around(Space::Single))
    .token(TokenRule::new(";").before(Space::None))
    .token(TokenRule::new("(").after(Space::None))
    .token(TokenRule::new(")").before(Space::None))
    .node(NodeRule::new("stmt").before(Space::Hard))
    .node(NodeRule::new("binary").group().indent(Indent::Hanging))
    .node(NodeRule::new("prefix").token(TokenRule::any().before(Space::None).after(Space::None)));
for op in ["+", "-", "*", "/"] {
    rules = rules.token(TokenRule::new(op).before(Space::Single).after(Space::Line));
}
let style = rules.compile(|name| calc.kind(name))?;

let parse = calc.parse("let   x=-(1+2)*y;- - 3;");
assert_eq!(format(parse.tree(), parse.source(), &style, 80)?, "let x = -(1 + 2) * y;\n- -3;\n");
# Ok::<(), Box<dyn std::error::Error>>(())

Examples

Performance

The formatter is two linear passes over the tree, both iterative: one flattens the tree into an event list (checking it against the source and settling list delimiters), one builds a single pretty_lang::Doc, merging runs of tokens and spaces into one text node, and pretty-lang renders it in one linear pass. Rule lookups are a binary search over rules sorted by kind, once per node and once per token.

Measured with the benchmarks in benches/, Windows x86_64, Rust stable, release profile. Parsing is done once, outside the measurement; the numbers are the formatter alone:

These are first numbers, not tuned ones, and they were taken on a machine running other builds, so expect ±20%. Throughput falls as inputs grow. A phase split at 792k nodes (one run, same input) showed roughly 15% flattening, 45% building the Doc, 15% rendering it, and 25% dropping it: most of the cost is allocating and freeing pretty-lang's reference-counted document nodes (one per text run, break, group, and join). The Doc in pretty-lang 1.x has no arena, so that is the next thing to change for speed.

cargo bench --bench bench

Limits

• Whether two tokens may touch is decided by a conservative heuristic (can_touch) unless the style supplies an exact test.
• Tokens whose extent depends on the whitespace after them (an unterminated string ends at the line break) need their regions passed to format_keeping; without them, removing that line break changes the token.
• A parser's recovery that leaves no trace in the tree is invisible to the formatter; trailing-separator edits skip lists with error nodes or unclosed delimiters, but Trailing::Preserve is the policy that never edits.
• In languages whose line breaks are tokens, rules that break lines add tokens.
• Indentation is spaces; widths count chars, not display columns.
• Range formatting (for LSP rangeFormatting) and a trailing comma "only when broken" are planned (see the ROADMAP).

Contributing

See dev/DIRECTIVES.md for engineering standards and the definition of done. Before a PR: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-features must be clean.