Crate loess

Source
Expand description
README / Examples (click to expand)

§Loess

Loess is a parser library and parser generator for proc macros.

For a simple but representative example of using Loess, see the inline-json5 crate.

Premade Rust grammar implementations can be found in loess-rust, with additional temporary wrappers available in loess-rust-opaque.

Here’s what to expect:

  • Fast builds. Loess’s core is compact and language agnostic. Direct grammar implementations like loess-rust should compile fairly quickly too.

  • A simple, flexible API. Loess is relatively unopinionated about how or what you parse, and you can construct (and destructure) Input at any time.

  • Really good error reporting from proc macros implemented with Loess, by default.

    Many parsing errors are easily or automatically recoverable, which means multiple errors can be reported at once while preserving as much regular output as possible (which means no or much fewer cascading errors!).

    Panics inside your parser can also be caught and reported as located errors with the original panic message.

  • A reasonably powerful parser-generator.

    grammar! can emit documentation (for enums) and PeekFrom, PopFrom and IntoTokens implementations on grammar types in general.

  • Powerful quote_into macros that expand efficiently and are language-agnostic.

    You can cleanly loop and/or branch within the template.

  • Low-allocation workflow.

    Loess can (usually) move tokens from input to output without cloning them. (You can still clone all included grammar types explicitly, including when pasting in quotes.)

Here’s what not to expect:

  • A general Syn-replacement (at least not soon).

    Loess is mainly aimed at implementing domain-specific languages that may cite fragments of Rust verbatim in their output. There is currently no focus on parsing Rust code or transforming it in-depth.

§Using $crate for full caller independence

loess::IntoTokens-methods take an (optionally empty) root: &TokenStream parameter, which all emitted fully qualified paths should be prefixed with.

In combination with a wrapper crate: This achieves full isolation regarding caller dependencies:

(click to expand code blocks)
// wrapper crate

#[macro_export]
macro_rules! my_macro {
    ($($tt:tt)*) => ( $crate::__::my_macro!([$crate] $($tt)*) );
}

#[doc(hidden)]
pub mod __ {
    pub use core; // Expected by `Errors`.
    pub use my_macro_impl::my_macro;
}
// my_macro_impl (proc macro)

use loess::{
    grammar, parse_once, parse_all,
    Errors, Input, IntoTokens,
    rust_grammar::{SquareBrackets},
};
use proc_macro2::{Span, TokenStream, TokenTree};

// […]

fn macro_impl(input: TokenStream) -> TokenStream {
    let mut input = Input {
        tokens: input.into_iter().collect(),
        end: Span::call_site(),
    };
    let mut errors = Errors::new();

    // `root` is implicitly a `TokenStream`.
    let Ok(SquareBrackets { contents: root, .. }) = parse_once(
            &mut input,
            &mut errors,
        ) else { return errors.collect_tokens(&TokenStream::new()) };

    grammar! {
        /// This represents your complete input grammar.
        /// This here is a placeholder, so it's empty.
        struct Grammar: PopFrom {}
    }

    // Checks for exhaustiveness.
    let parsed = parse_all(&mut input, &mut errors).next();
    let mut output = errors.collect_tokens(&root);

    if let Some(Grammar {}) = parsed {
        // Emit your output here.
    }

    output
}

Loess is a parser library and parser generator for proc macros.

For a simple but representative example of using Loess, see the inline-json5 crate.

In most cases you’ll want to:

  1. generate custom grammar implementations with grammar! (You can also easily implement parts manually.),
  2. create (mutable) instances of Input and Errors,
  3. step through the input with parse_once, parse_once_with and/or parse_once_with_infallible,
  4. consume the last of the input with parse_all, parse_all_with or parse_all_with_infallible,
  5. perform any fallible transforms you need, possibly pushing more Errors into your Errors,
  6. if errors is your Errors, have let mut output: proc_macro2::TokenStream = errors.collect_tokens(); convert it into the start of your output,
  7. emit your regular output with quote_into_mixed_site! (recommended), quote_into_with_exact_span! or quote_into_call_site!, which accept interpolation and control flow directives.

You can call either Iterator::collect (for repeats) or Iterator::next (for one value) on step 3. Either way, the parsing iterator will check for unconsumed tokens remaining in the Input when dropped and report to the Errors accordingly.

You can combine step 2 into step 3 with a grammar!-generated top-level grammar, but for proc macros embedded in a runtime library, in most cases I recommend getting $crate from a wrapper macro_rules!-macro first. (See full example above.)

Some parsing errors are recoverable, but still translate to compile_error! calls being generated in step 5. Your macro should seamlessly continue to operate in such cases, which helps prevent noise from cascading errors due to e.g. missing items, making it much easier for your macro’s users to find problems with the input.

You can download a .code-snippets file for Loess’s macros and quote macro directives here: https://github.com/Tamschi/Asteracea/blob/develop/.vscode/Loess.code-snippets

§Features

None are default, as DSL macros might not need Rust’s grammar at all.

§"rust_grammar"

Enables rust_grammar.

§"opaque_rust_grammar" enables "rust_grammar", depends on syn and quote

Adds additional opaque Rust grammar tokens, to consume, paste and clone for example Statements and Patterns.

These preliminary implementations are Syn-based and can’t be inspected.

Modules§

error_priorities
ConstErrorPriority types for use with Exhaustive and EndOfInput.
rust_grammarDeprecated
With "rust_grammar": Tokens representing the stable Rust programming language, closely following The Rust Reference.

Macros§

grammar
Parser- and serialiser-generator macro.
quote_into_call_site
Like quote_into_mixed_site!, but resolved according to Span::call_site().
quote_into_mixed_site
Simple generic quotation (statement) macro that works well with Loess’s types.
quote_into_with_exact_span
Like quote_into_mixed_site!, but using $span directly for quoted tokens.
raw_quote_into_call_site
Like raw_quote_into_mixed_site!, but resolved according to Span::call_site().
raw_quote_into_mixed_site
Simple generic quotation (statement) macro that efficiently emits tokens verbatim.
raw_quote_into_with_exact_span
Like raw_quote_into_mixed_site!, but using $span directly for quoted tokens.

Structs§

Eager
Wraps a collection type to eagerly parse values that are PeekFrom, but to stop when PopFrom::peek_pop_from returns None.
EndOfInput
Fails to parse and emits an Error with the given ConstErrorPriority for any unconsumed tokens in Input.
Error
A Span-located proc macro error with ErrorPriority.
Usually submitted through Errors::push.
ErrorPriority
An opaque Error priority.
Errors
A collection of Errors submitted during e.g. parsing with PopFrom.
Exhaustive
Doesn’t fail to parse but emits an Error with the given ConstErrorPriority for any unconsumed tokens in Input after T.
HandledPanic
A substitute panic that isn’t reported as Error. (Read for panic handling info!)
Input
Input tokens with end-Span.
For use with PeekFrom and PopFrom.

Traits§

ConstErrorPriority
ErrorPriority as generic type argument.
IntoTokens
Spreads self into its contained or representative TokenTrees.
PeekFrom
Determines if Self may be be parseable from an Input.
This is often a cursory check!
PopFrom
Consumes from Input to create Result<Self, ()> and emit to Errors.
SimpleSpanned
Has a single Span.

Functions§

parse_all
Conveniently parses remaining Input through PopFrom, catching and submitting panics to the given Errors.
parse_all_with
Parses remaining Input through FnMut, catching and submitting panics to the given Errors.
parse_all_with_infallible
Low-level function that parses remaining Input through FnMut without also catching Err(()), catching and submitting panics to the given Errors.
parse_once
Convenient non-repeating PopFrom::pop_from-unwind-catcher that reports panics to the given Errors.
parse_once_with
FnOnce-unwind-catcher that reports panics to the given Errors.
parse_once_with_infallible
Low-level FnOnce-unwind-catcher that reports panics to the given Errors without also catching Err(()).