Expand description
Syn is a parsing library for parsing a stream of Rust tokens into a syntax tree of Rust source code.
Currently this library is geared toward use in Rust procedural macros, but contains some APIs that may be useful more generally.
-
Data structures — Syn provides a complete syntax tree that can represent any valid Rust source code. The syntax tree is rooted at
syn::Filewhich represents a full source file, but there are other entry points that may be useful to procedural macros includingsyn::Item,syn::Exprandsyn::Type. -
Derives — Of particular interest to derive macros is
syn::DeriveInputwhich is any of the three legal input items to a derive macro. An example below shows using this type in a library that can derive implementations of a user-defined trait. -
Parsing — Parsing in Syn is built around parser functions with the signature
fn(ParseStream) -> Result<T>. Every syntax tree node defined by Syn is individually parsable and may be used as a building block for custom syntaxes, or you may dream up your own brand new syntax without involving any of our syntax tree types. -
Location information — Every token parsed by Syn is associated with a
Spanthat tracks line and column information back to the source of that token. These spans allow a procedural macro to display detailed error messages pointing to all the right places in the user’s code. There is an example of this below. -
Feature flags — Functionality is aggressively feature gated so your procedural macros enable only what they need, and do not pay in compile time for all the rest.
Example of a derive macro
The canonical derive macro using Syn looks like this. We write an ordinary
Rust function tagged with a proc_macro_derive attribute and the name of
the trait we are deriving. Any time that derive appears in the user’s code,
the Rust compiler passes their data structure as tokens into our macro. We
get to execute arbitrary Rust code to figure out what to do with those
tokens, then hand some tokens back to the compiler to compile into the
user’s crate.
[dependencies]
syn = "1.0"
quote = "1.0"
[lib]
proc-macro = trueuse proc_macro::TokenStream;
use quote::quote;
use syn::{parse_macro_input, DeriveInput};
#[proc_macro_derive(MyMacro)]
pub fn my_macro(input: TokenStream) -> TokenStream {
// Parse the input tokens into a syntax tree
let input = parse_macro_input!(input as DeriveInput);
// Build the output, possibly using quasi-quotation
let expanded = quote! {
// ...
};
// Hand the output tokens back to the compiler
TokenStream::from(expanded)
}The heapsize example directory shows a complete working implementation
of a derive macro. It works on any Rust compiler 1.31+. The example derives
a HeapSize trait which computes an estimate of the amount of heap memory
owned by a value.
pub trait HeapSize {
/// Total number of bytes of heap memory owned by `self`.
fn heap_size_of_children(&self) -> usize;
}The derive macro allows users to write #[derive(HeapSize)] on data
structures in their program.
#[derive(HeapSize)]
struct Demo<'a, T: ?Sized> {
a: Box<T>,
b: u8,
c: &'a str,
d: String,
}Spans and error reporting
The token-based procedural macro API provides great control over where the
compiler’s error messages are displayed in user code. Consider the error the
user sees if one of their field types does not implement HeapSize.
#[derive(HeapSize)]
struct Broken {
ok: String,
bad: std::thread::Thread,
}By tracking span information all the way through the expansion of a
procedural macro as shown in the heapsize example, token-based macros in
Syn are able to trigger errors that directly pinpoint the source of the
problem.
error[E0277]: the trait bound `std::thread::Thread: HeapSize` is not satisfied
--> src/main.rs:7:5
|
7 | bad: std::thread::Thread,
| ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `HeapSize` is not implemented for `Thread`Parsing a custom syntax
The lazy-static example directory shows the implementation of a
functionlike!(...) procedural macro in which the input tokens are parsed
using Syn’s parsing API.
The example reimplements the popular lazy_static crate from crates.io as a
procedural macro.
lazy_static! {
static ref USERNAME: Regex = Regex::new("^[a-z0-9_-]{3,16}$").unwrap();
}The implementation shows how to trigger custom warnings and error messages on the macro input.
warning: come on, pick a more creative name
--> src/main.rs:10:16
|
10 | static ref FOO: String = "lazy_static".to_owned();
| ^^^Testing
When testing macros, we often care not just that the macro can be used
successfully but also that when the macro is provided with invalid input it
produces maximally helpful error messages. Consider using the trybuild
crate to write tests for errors that are emitted by your macro or errors
detected by the Rust compiler in the expanded code following misuse of the
macro. Such tests help avoid regressions from later refactors that
mistakenly make an error no longer trigger or be less helpful than it used
to be.
Debugging
When developing a procedural macro it can be helpful to look at what the
generated code looks like. Use cargo rustc -- -Zunstable-options --pretty=expanded or the cargo expand subcommand.
To show the expanded code for some crate that uses your procedural macro,
run cargo expand from that crate. To show the expanded code for one of
your own test cases, run cargo expand --test the_test_case where the last
argument is the name of the test file without the .rs extension.
This write-up by Brandon W Maister discusses debugging in more detail: Debugging Rust’s new Custom Derive system.
Optional features
Syn puts a lot of functionality behind optional features in order to optimize compile time for the most common use cases. The following features are available.
derive(enabled by default) — Data structures for representing the possible input to a derive macro, including structs and enums and types.full— Data structures for representing the syntax tree of all valid Rust source code, including items and expressions.parsing(enabled by default) — Ability to parse input tokens into a syntax tree node of a chosen type.printing(enabled by default) — Ability to print a syntax tree node as tokens of Rust source code.visit— Trait for traversing a syntax tree.visit-mut— Trait for traversing and mutating in place a syntax tree.fold— Trait for transforming an owned syntax tree.clone-impls(enabled by default) — Clone impls for all syntax tree types.extra-traits— Debug, Eq, PartialEq, Hash impls for all syntax tree types.proc-macro(enabled by default) — Runtime dependency on the dynamic library libproc_macro from rustc toolchain.
Modules
parsingparsingfoldparsingparsing and printingSpan of the complete contents of a syntax
tree node.visitvisit-mutMacros
parsingparsingparsingparsing and proc-macroparsing and printingquote! macro but uses
type inference to figure out a return type for those tokens.parsing and printingparse_quote! + quote_spanned!.Structs
full or deriveextern "C".full or derive<K, V> in HashMap<K, V>.fullmatch expression: 0...10 => { return true; }.full or derive#[repr(transparent)].full or deriveusize in fn(usize) -> bool.full or deriveItem = u8.fullfull or derivefor<'a, 'b, 'c>.full or deriveconst LENGTH: usize.full or deriveIterator<Item: Display>.deriveproc_macro_derive macro.deriveproc_macro_derive macro.deriveproc_macro_derive macro.deriveproc_macro_derive macro.full[a, b, c, d].fulla = compute().fullcounter += 1.fullasync { ... }.fullfut.await.full or derivea + b, a * b.full{ ... }.fullbox f.fullbreak, with an optional label to break and an optional
expression.full or deriveinvoke(a, b).full or derivefoo as f64.full|a, b| a + b.fullcontinue, with an optional label.full or deriveobj.k) or unnamed tuple struct
field (obj.0).fullfor pat in expr { ... }.fullfullif expression with an optional else block: if expr { ... } else { ... }.full or derivevector[2].fulllet guard: let Some(x) = opt.full or derive1, "foo".fullloop { ... }.fullformat!("{}", q).fullmatch expression: match n { Some(n) => {}, None => {} }.fullx.foo::<T>(a, b).full or derive(a + b).full or derivestd::mem::replace possibly containing generic
parameters and a qualified self-type.full1..2, 1.., ..2, 1..=2, ..=2.full&a or &mut a.full[0u8; N].fullreturn, with an optional value to be returned.fullPoint { x: 1, y: 1 }.fullexpr?.fulltry { ... }.full(a, b, c, d).fullfoo: f64.full or derive!x, *x.fullunsafe { ... }.fullwhile expr { ... }.fullyield expr.full or derivefullfullfull or derivePoint { x: f64, y: f64 }.full or deriveSome(T).fullfullextern block.fullextern block: static ext: u8.fullextern block: type void.full or derivefull or derive) and printingGenerics::split_for_impl.fullfullfullfullfull or derivefullconst MAX: u16 = 65535.fullenum Foo<A, B> { A(A), B(B) }.fullextern crate item: extern crate serde.fullfn process(n: usize) -> Result<()> { ... }.fullextern "C" { ... }.fullimpl<A> Trait for Data<A> { ... }.fullmacro_rules! definitions.fullmacro keyword.fullmod m or mod m { ... }.fullstatic BIKE: Shed = Shed(42).fullstruct Foo<A> { x: A }.fullpub trait Iterator { ... }.fullpub trait SharableIterator = Iterator + Sync.fulltype Result<T> = std::result::Result<T, MyError>.fullunion Foo<A, B> { x: A, y: B }.fulluse std::collections::HashMap.fullfor, while, or loop.'a.full or derive'a: 'b + 'c + 'd.true or false.b'f'.b"foo".'a'.1f64 or 1.0e10f64.1 or 1u16."foo".fulllet binding: let x: u64 = s.parse()?.full or deriveprintln!("{}", mac).full or derivederive(Copy, Clone).full or derivefeature = "nightly".full::<> explicit type parameters passed to a method call:
parse::<u64>().full or derive(A, B) -> C in Fn(A,B) -> C.fullbox v.fullref mut binding @ SUBPATTERN.full0.fullfullfullColor::Red, optionally qualified with a
self-type.full1..=2.full&mut var.full[0, 1, ..]full[a, b, ref i @ .., y, z].fullVariant { x, y, .. }.full(a, b).fullVariant(x, y, .., z).fullfoo: f64.full_.full or derivestd::collections::HashMap).full or derivefull or derivewhere clause (unsupported).full or derivewhere clause: 'a: 'b + 'c.full or derivewhere clause: for<'c> Foo<'c>: Trait<'c>.full or deriveT in <T as Display>::fmt.fullself argument of an associated method, whether taken by value
or by reference.fullunsafe fn initialize(&self).full or derivefullfullfullfullfull or derive) and printingTypeGenerics::as_turbofish.full or derive[T; n].full or derivefn(usize) -> bool.full or derive) and printingGenerics::split_for_impl.full or derivefull or deriveimpl Bound1 + Bound2 + Bound3 type where Bound is a trait or
a lifetime.full or derive_.full or derivefull or derive!.full or deriveT: Into<String>.full or derivefull or derivestd::slice::Iter, optionally qualified with a
self-type as in <Vec<T> as SomeTrait>::Associated.full or derive*const T or *mut T.full or derive&'a T or &'a mut T.full or derive[T].full or derivedyn Bound1 + Bound2 + Bound3 where Bound is a
trait or a lifetime.full or derive(A, B, C, String).fulluse item: *.fulluse item: {A, B, C}.fulluse item: HashMap.fulluse item: std::....fulluse item: HashMap as Map.full or derivefull or derivefull or derivecrate.full or derivepub.full or derivepub(self) or
pub(super) or pub(crate) or pub(in some::module).full or derivewhere clause in a definition: where T: Deserialize<'de>, D: 'static.Enums
full or derivefull or derive+, +=, &.derivefull or derivefull or derivefulln: usize in fn f(n: usize).fullextern block.full or derive'a, T, or Item = T.T.full or deriveT: Into<String>,
'a: 'b, const LEN: usize.fullfullfull or derivem!(...) or m!{...} or m![...].full or derivefull or derivefull or derivefullfull or derivefullfull or derivefullr##"data"##.full or derive? in
?Sized.fullfull or derivefull or derivefull or derive*, !, -.fulluse item: Type as Renamed or *.full or derivepub or
pub(restricted).full or derivewhere clause: T: Deserialize<'de>.Functions
parsing and proc-macroparsingparsing and fullparsingType Definitions
full or derive