macroex

An extractor based low level macro parsing crate that provides high level parsing support through derive macros.

FromMacro and Extractor

[FromMacro] is the bread and butter of this crate. FromMacro provides a from_one and a from_many function that parses a [TokenTree] and a [TokenStream] respectively.

We can mostly assume from_many will contain two or more TokenTrees. If not that is considered a bug in an extractor.

All implementors of FromMacro are Extractors of one TokenTree. When used on a TokenStream iterator directly, they will consume a single TokenTree and try to parse it using from_one.

let mut iter = quote!{45, Hello; true false}.into_iter();
let a: i32 = iter.extract()?;
let b: Punct = iter.extract()?;
// Extracts a string value from an ident
let IdentString(c) = iter.extract()?;
// Validates a semicolon
let d: PunctOf<';'> = iter.extract()?;
let e: bool = iter.extract()?;
// Validates a false literal
let f: LitFalse = iter.extract()?;

This is pretty great! As most things can be represented as a single [TokenTree].

// This is a single TokenTree::Group
{
    name: "Tom".
    age: 45,
    children: [
        "Tim", "Tam"
    ],
}

However there are other things one TokenTree cannot account for.

// This fails because -45 is two tokens
let a: i32 = quote!{-45}.into_iter().extract().unwrap();

Wrapping [FromMacro] implementers in other Extractors allow FromMacro implementors to parse additional TokenTrees and potentially utilize the from_many method if more than one TokenTree is matched.

// Note -45 is two tokens
let mut iter = quote!{-45}.into_iter();
// All extracts everything from a stream
let All(a) = iter.extract()?;
assert_eq!(a, -45i32);

let mut iter = quote!{-45, 21, 9.5,}.into_iter();
// CommaExtractor extracts until a comma or end of stream is found.
let CommaExtractor(a) = iter.extract()?;
let CommaExtractor(b) = iter.extract()?;
let CommaExtractor(c) = iter.extract()?;
// EndOfStream is a unit struct extractor and this asserts iter is empty
let EndOfStream = iter.extract()?;
assert_eq!(a, -45);
assert_eq!(b, 21);
assert_eq!(c, 9.5);

Derive

We provide derive macro FromMacro and [FromAttrs] that functions similarly to to serde::Deserialize. This enables ergonomic parsing for structs and enums following a specific data format.

FromMacro parses syntax similar to native rust, while [FromAttrs] parses syntax commonly used in macro attributes.

Why not serde_tokenstream?

Since we do not use the serde data model. Our data model is much more powerful in the macro context. We are allowed to extract all [FromMacro] implementors, including [TokenStream], Ident, Group, etc.

FromMacro

FromMacro parses syntax similar to native rust.

Examples

Use Case

Since we are likely to be parsing configurations in macros, we supply a Default::default() value if a field is not found.

You are required to opt out of this with #[macroex(required)] if your type does not implement [Default].

#[derive(FromMacro)]
pub struct Person {
    pub name: String,
    pub age: i32,
    pub height: f32,
    // This works as long as Gender implements `FromMacro` and `Default`
    pub gender: Gender,
    // Using Option is idiomatic to handle the default case.
    pub hair_color: Option<NumberList<[f32;4]>>,
    // We can extract macro based things
    pub do_something: Option<TokenStream>,
}

Example macro input:

person! {
    name: "Lina",
    age: 23,
    gender: Female,
    hair_color: [0.7, 0.4, 0],
    do_something: {
        let w = "world";
        println!("Hello, {}!", w)
    },
}

Attributes

The [FromMacro] macro supports the following attributes:

#[derive(FromMacro)]
// We use the same casing names as serde.
#[macroex(rename_all="SCREAMING-KEBAB-CASE")]
pub struct Animal {
    // Errors if not specified.
    #[macroex(required)]
    pub name: String,
    // Evaluate an expression instead of `Default::default()`
    #[macroex(default="0.0")]
    pub height: f32,
    #[macroex(default=r#""dog".to_owned()"#)]
    pub species: String,
    // Take strings as inputs, and collects them into a vec.
    #[macroex(repeat)]
    // and rename "nicknames" to "nickname" during parsing.
    #[macroex(rename="nickname")]
    pub nicknames: Vec<String>,
}

FromAttrs

[FromAttrs] Generates a simple [FromMacro] implementation for syntax commonly associated with macro attributes.

This macro is only allowed on named structs and supports 3 basic syntax:

• .., name, .. parses to name: true, which matches a boolean value.
• .., name = expr, .. parses to name: expr
• .., name(..), .. parses to name: T{ .. }

Other types like fieldless enums can potentially use FromMacro to generated compatible [FromMacro] implementations to use with this macro.

Example

We use the same set of attributes as [FromMacro]

#[derive(FromAttrs)]
#[macroex(rename_all="snake_case")]
pub struct Animal {
    #[macroex(required)]
    pub name: String,
    #[macroex(default="0.0")]
    pub height: f32,
    #[macroex(default=r#"Ident::new("Dog", Span::call_site())"#)]
    pub species: Ident,
    #[macroex(repeat, rename="mascot")]
    pub mascot_infos: Vec<MascotInfo>,
}

Example attribute:

#[animal(name = "Ferris", species = Crab, mascot(language = "Rust"))]

We can parse either

(name = "Ferris", species = Crab, mascot(language = "Rust"))

with from_one, or

name = "Ferris", species = Crab, mascot(language = "Rust")

with from_many, commonly extracted with syn.

Macro Chaining and Hygeine

We treat our input as string-like and we will try to flatten all None delimited groups encountered during parsing.