Rust Quasiquoter

This crate implements the quote! macro as a procedural macro, instead of the original quote! macro, implemented with macro_rules!.

The quote! macro turns Rust syntax tree data structures into tokens of source code.

Procedural macros in Rust receive a stream of tokens as input, execute arbitrary Rust code to determine how to manipulate those tokens, and produce a stream of tokens to hand back to the compiler to compile into the caller's crate. Quasi-quoting is a solution to one piece of that -- producing tokens to return to the compiler.

The idea of quasi-quoting is that we write code that we treat as data. Within the quote! macro, we can write what looks like code to our text editor or IDE. We get all the benefits of the editor's brace matching, syntax highlighting, indentation, and maybe autocompletion. But rather than compiling that as code into the current crate, we can treat it as data, pass it around, mutate it, and eventually hand it back to the compiler as tokens to compile into the macro caller's crate.

This crate is motivated by the procedural macro use case, but it is a general-purpose Rust quasi-quoting library and is not specific to procedural macros.

Why use proc-quote?

This crate serves the same purpose as quote however it is implemented with procedural macros rather than macro_rules!. Switching from quote to the proc_quote crate should not require any change in the code in most cases.

Besides, opting to proc_quote has the advantage of lifting some of the limitations of the original quasi-quoting crate:

• Interpolate a ToTokens variable inside of a repeating block
• Interpolate the same Iterator more than once inside of the same repeating block
• Use Vec, arrays and slices in a repeating block without consuming the variable

Interpolate a ToTokens variable inside of a repeating block

In proc_quote, this is now possible:

let i = 1..5; // Iterator<ToTokens> 
let a = "a"; // ToTokens

let q = quote! {
    #(#i #a)*
};
assert_eq!("1i32 \"a\" 2i32 \"a\" 3i32 \"a\" 4i32 \"a\"", q.to_string());

You may interpolate variables implementing ToTokens inside repeating blocks. The variable will be interpolated in every iteration.

Interpolate the same Iterator more than once inside of the same repeating block

In the original quote! macro, the same iterator could not be interpolated more than once in repeating blocks. In other words, for a: impl Iterator<ToTokens>, quote!( #(#a #a)* ) didn't work.

This is fixed in proc-quote.

Use Vec, arrays and slices in a repeating block without consuming the variable

In proc-quote, any type that implements Borrow<[T]> inside a repeating pattern will iterate without consuming the variable (using the .iter method). This includes Vec, arrays and slices.

This means you no longer need to create ten different variables to iterate the same elements in ten different places in your quote! macros.

From quote to proc-quote

If you are already using the quote crate but you want to leverage the advantages of proc-quote, you can easily switch crates.

After changing your Cargo.toml dependency, change the following:

extern crate quote;
use quote::quote;
use quote::quote_spanned;

respectively into:

extern crate proc_quote;
use proc_quote::quote;
use proc_quote::quote_spanned;

Note: You may also remove #![recursion_limit="..."] as it is not needed in the procedural macro.

And that's basically it, most of the times!

However, repeating patterns #(...)* do NOT work directly with IntoIterator in proc-quote (see Repeat trait). If your type was already an Iterator or any type implementing Borrow<[T]> (such as Vec, slices, arrays, ...), your code will work the same as in quote.

On the other hand, cases like TokenStream (implements IntoIterator but it is not any of the cases stated above) must be explicitely turned into iterators before being interpolated. That is, if you had:

let a: TokenStream = /* ... */; 
let q = quote!(#(#a)*);

You must change it to:

let a: TokenStream = /* ... */;
let a = a.into_iter(); 
let q = quote!(#(#a)*);

This explicitness is necessary because of proc-quote's new features. In this example, TokenStream would have be seen as a type implementing ToTokens instead of an Iterator of tokens, which makes sense because it is much more frequently used as the former when interpolated.

Syntax

The quote crate provides a quote! macro within which you can write Rust code that gets packaged into a TokenStream and can be treated as data. You should think of TokenStream as representing a fragment of Rust source code.

Within the quote! macro, interpolation is done with #var. Any type implementing the quote::ToTokens trait can be interpolated. This includes most Rust primitive types as well as most of the syntax tree types from syn.

let tokens = quote! {
    struct SerializeWith #generics #where_clause {
        value: &'a #field_ty,
        phantom: core::marker::PhantomData<#item_ty>,
    }

    impl #generics serde::Serialize for SerializeWith #generics #where_clause {
        fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
        where
            S: serde::Serializer,
        {
            #path(self.value, serializer)
        }
    }

    SerializeWith {
        value: #value,
        phantom: core::marker::PhantomData::<#item_ty>,
    }
};

Repetition

Repetition is done using #(...)* or #(...),* similar to macro_rules!. This iterates through the elements of any variable interpolated within the repetition and inserts a copy of the repetition body for each one.

• #(#var)* — no separators
• #(#var),* — the character before the asterisk is used as a separator
• #( struct #var; )* — the repetition can contain other things
• #( #k => println!("{}", #v), )* — even multiple interpolations
• #(let #var = self.#var;)* - the same variable can be used more than once

Note that there is a difference between #(#var ,)* and #(#var),*—the latter does not produce a trailing comma. This matches the behavior of delimiters in macro_rules!.

The proc_quote::Repeat trait defines which types are allowed to be interpolated inside a repition pattern.

Which types do Repeat:

• Iterator<T> consumes the iterator, iterating through every element.
• Borrow<[T]> (includes Vec, array, and slice) iterates with the slice::iter method, thus not consuming the original data.
• ToTokens, interpolates the variable in every iteration.

Which types do NOT Repeat:

• IntoIterator, to avoid ambiguity (Ex. "Which behavior would have been used for Vec, which implements both IntoIterator and Borrow<[T]>?"; "Which behavior would have been used for TokenStream, which implements both IntoIterator and ToTokens?"). To use the iterator, you may call IntoIterator::into_iter explicitly.
• Ambiguous types that implement at least two of the Repeat traits. In the very unlikely case this happens, disambiguate the type by wrapping it under some structure that only implements the trait you desire to use.

Returning tokens to the compiler

The quote! macro evaluates to an expression of type proc_macro2::TokenStream. Meanwhile Rust procedural macros are expected to return the type proc_macro::TokenStream.

The difference between the two types is that proc_macro types are entirely specific to procedural macros and cannot ever exist in code outside of a procedural macro, while proc_macro2 types may exist anywhere including tests and non-macro code like main.rs and build.rs. This is why even the procedural macro ecosystem is largely built around proc_macro2, because that ensures the libraries are unit testable and accessible in non-macro contexts.

There is a From-conversion in both directions so returning the output of quote! from a procedural macro usually looks like tokens.into() or proc_macro::TokenStream::from(tokens).

Examples

Combining quoted fragments

Usually you don't end up constructing an entire final TokenStream in one piece. Different parts may come from different helper functions. The tokens produced by quote! themselves implement ToTokens and so can be interpolated into later quote! invocations to build up a final result.

let type_definition = quote! {...};
let methods = quote! {...};

let tokens = quote! {
    #type_definition
    #methods
};

Constructing identifiers

Suppose we have an identifier ident which came from somewhere in a macro input and we need to modify it in some way for the macro output. Let's consider prepending the identifier with an underscore.

Simply interpolating the identifier next to an underscore will not have the behavior of concatenating them. The underscore and the identifier will continue to be two separate tokens as if you had written _ x.

// incorrect
quote! {
    let mut _#ident = 0;
}

The solution is to perform token-level manipulations using the APIs provided by Syn and proc-macro2.

let concatenated = format!("_{}", ident);
let varname = syn::Ident::new(&concatenated, ident.span());
quote! {
    let mut #varname = 0;
}

Making method calls

Let's say our macro requires some type specified in the macro input to have a constructor called new. We have the type in a variable called field_type of type syn::Type and want to invoke the constructor.

// incorrect
quote! {
    let value = #field_type::new();
}

This works only sometimes. If field_type is String, the expanded code contains String::new() which is fine. But if field_type is something like Vec<i32> then the expanded code is Vec<i32>::new() which is invalid syntax. Ordinarily in handwritten Rust we would write Vec::<i32>::new() but for macros often the following is more convenient.

quote! {
    let value = <#field_type>::new();
}

This expands to <Vec<i32>>::new() which behaves correctly.

A similar pattern is appropriate for trait methods.

quote! {
    let value = <#field_type as core::default::Default>::default();
}

Hygiene

Any interpolated tokens preserve the Span information provided by their ToTokens implementation. Tokens that originate within a quote! invocation are spanned with Span::call_site().

A different span can be provided explicitly through the quote_spanned! macro.

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.