Crate syn

  

Syn is a parsing library for parsing a stream of Rust tokens into a syntax tree of Rust source code.

Currently this library is geared toward use in Rust procedural macros, but contains some APIs that may be useful more generally.

• Data structures — Syn provides a complete syntax tree that can represent any valid Rust source code. The syntax tree is rooted at syn::File which represents a full source file, but there are other entry points that may be useful to procedural macros including syn::Item, syn::Expr and syn::Type.

• Derives — Of particular interest to derive macros is syn::DeriveInput which is any of the three legal input items to a derive macro. An example below shows using this type in a library that can derive implementations of a user-defined trait.

• Parsing — Parsing in Syn is built around parser functions with the signature fn(ParseStream) -> Result<T>. Every syntax tree node defined by Syn is individually parsable and may be used as a building block for custom syntaxes, or you may dream up your own brand new syntax without involving any of our syntax tree types.

• Location information — Every token parsed by Syn is associated with a Span that tracks line and column information back to the source of that token. These spans allow a procedural macro to display detailed error messages pointing to all the right places in the user's code. There is an example of this below.

• Feature flags — Functionality is aggressively feature gated so your procedural macros enable only what they need, and do not pay in compile time for all the rest.

Example of a derive macro

The canonical derive macro using Syn looks like this. We write an ordinary Rust function tagged with a proc_macro_derive attribute and the name of the trait we are deriving. Any time that derive appears in the user's code, the Rust compiler passes their data structure as tokens into our macro. We get to execute arbitrary Rust code to figure out what to do with those tokens, then hand some tokens back to the compiler to compile into the user's crate.

[dependencies]
syn = "1.0"
quote = "1.0"

[lib]
proc-macro = true

use proc_macro::TokenStream;
use quote::quote;
use syn::{parse_macro_input, DeriveInput};

#[proc_macro_derive(MyMacro)]
pub fn my_macro(input: TokenStream) -> TokenStream {
    // Parse the input tokens into a syntax tree
    let input = parse_macro_input!(input as DeriveInput);

    // Build the output, possibly using quasi-quotation
    let expanded = quote! {
        // ...
    };

    // Hand the output tokens back to the compiler
    TokenStream::from(expanded)
}

The heapsize example directory shows a complete working implementation of a derive macro. It works on any Rust compiler 1.31+. The example derives a HeapSize trait which computes an estimate of the amount of heap memory owned by a value.

pub trait HeapSize {
    /// Total number of bytes of heap memory owned by `self`.
    fn heap_size_of_children(&self) -> usize;
}

The derive macro allows users to write #[derive(HeapSize)] on data structures in their program.

#[derive(HeapSize)]
struct Demo<'a, T: ?Sized> {
    a: Box<T>,
    b: u8,
    c: &'a str,
    d: String,
}

Spans and error reporting

The token-based procedural macro API provides great control over where the compiler's error messages are displayed in user code. Consider the error the user sees if one of their field types does not implement HeapSize.

#[derive(HeapSize)]
struct Broken {
    ok: String,
    bad: std::thread::Thread,
}

By tracking span information all the way through the expansion of a procedural macro as shown in the heapsize example, token-based macros in Syn are able to trigger errors that directly pinpoint the source of the problem.

error[E0277]: the trait bound `std::thread::Thread: HeapSize` is not satisfied
 --> src/main.rs:7:5
  |
7 |     bad: std::thread::Thread,
  |     ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `HeapSize` is not implemented for `Thread`

Parsing a custom syntax

The lazy-static example directory shows the implementation of a functionlike!(...) procedural macro in which the input tokens are parsed using Syn's parsing API.

The example reimplements the popular lazy_static crate from crates.io as a procedural macro.

lazy_static! {
    static ref USERNAME: Regex = Regex::new("^[a-z0-9_-]{3,16}$").unwrap();
}

The implementation shows how to trigger custom warnings and error messages on the macro input.

warning: come on, pick a more creative name
  --> src/main.rs:10:16
   |
10 |     static ref FOO: String = "lazy_static".to_owned();
   |                ^^^

Testing

When testing macros, we often care not just that the macro can be used successfully but also that when the macro is provided with invalid input it produces maximally helpful error messages. Consider using the trybuild crate to write tests for errors that are emitted by your macro or errors detected by the Rust compiler in the expanded code following misuse of the macro. Such tests help avoid regressions from later refactors that mistakenly make an error no longer trigger or be less helpful than it used to be.

Debugging

When developing a procedural macro it can be helpful to look at what the generated code looks like. Use cargo rustc -- -Zunstable-options --pretty=expanded or the cargo expand subcommand.

To show the expanded code for some crate that uses your procedural macro, run cargo expand from that crate. To show the expanded code for one of your own test cases, run cargo expand --test the_test_case where the last argument is the name of the test file without the .rs extension.

This write-up by Brandon W Maister discusses debugging in more detail: Debugging Rust's new Custom Derive system.

Optional features

Syn puts a lot of functionality behind optional features in order to optimize compile time for the most common use cases. The following features are available.

• derive (enabled by default) — Data structures for representing the possible input to a derive macro, including structs and enums and types.
• full — Data structures for representing the syntax tree of all valid Rust source code, including items and expressions.
• parsing (enabled by default) — Ability to parse input tokens into a syntax tree node of a chosen type.
• printing (enabled by default) — Ability to print a syntax tree node as tokens of Rust source code.
• visit — Trait for traversing a syntax tree.
• visit-mut — Trait for traversing and mutating in place a syntax tree.
• fold — Trait for transforming an owned syntax tree.
• clone-impls (enabled by default) — Clone impls for all syntax tree types.
• extra-traits — Debug, Eq, PartialEq, Hash impls for all syntax tree types.
• proc-macro (enabled by default) — Runtime dependency on the dynamic library libproc_macro from rustc toolchain.

Modules

buffer	parsing A stably addressed token buffer supporting efficient traversal based on a cheaply copyable cursor.
ext	parsing Extension traits to provide parsing methods on foreign types.
fold	fold Syntax tree traversal to transform the nodes of an owned syntax tree.
parse	parsing Parsing interface for parsing a token stream into a syntax tree node.
punctuated	A punctuated sequence of syntax tree nodes separated by punctuation.
spanned	parsing and printing A trait that can provide the Span of the complete contents of a syntax tree node.
token	Tokens representing Rust punctuation, keywords, and delimiters.
visit	visit Syntax tree traversal to walk a shared borrow of a syntax tree.
visit_mut	visit-mut Syntax tree traversal to mutate an exclusive borrow of a syntax tree in place.

Macros

Token	A type-macro that expands to the name of the Rust type representation of a given token.
braced	parsing Parse a set of curly braces and expose their content to subsequent parsers.
bracketed	parsing Parse a set of square brackets and expose their content to subsequent parsers.
custom_keyword	Define a type that supports parsing and printing a given identifier as if it were a keyword.
custom_punctuation	Define a type that supports parsing and printing a multi-character symbol as if it were a punctuation token.
parenthesized	parsing Parse a set of parentheses and expose their content to subsequent parsers.
parse_macro_input	parsing and proc-macro Parse the input TokenStream of a macro, triggering a compile error if the tokens fail to parse.
parse_quote	parsing and printing Quasi-quotation macro that accepts input like the quote! macro but uses type inference to figure out a return type for those tokens.

Structs

Abi	full or derive The binary interface of a function: extern "C".
AngleBracketedGenericArguments	full or derive Angle bracketed arguments of a path segment: the <K, V> in HashMap<K, V>.
Arm	full One arm of a match expression: 0...10 => { return true; }.
Attribute	full or derive An attribute like #[repr(transparent)].
BareFnArg	full or derive An argument in a function type: the usize in fn(usize) -> bool.
Binding	full or derive A binding (equality constraint) on an associated type: Item = u8.
Block	full A braced block containing Rust statements.
BoundLifetimes	full or derive A set of bound lifetimes: for<'a, 'b, 'c>.
ConstParam	full or derive A const generic parameter: const LENGTH: usize.
Constraint	full or derive An associated type bound: Iterator<Item: Display>.
DataEnum	derive An enum input to a proc_macro_derive macro.
DataStruct	derive A struct input to a proc_macro_derive macro.
DataUnion	derive An untagged union input to a proc_macro_derive macro.
DeriveInput	derive Data structure sent to a proc_macro_derive macro.
Error	Error returned when a Syn parser cannot parse the input tokens.
ExprArray	full A slice literal expression: [a, b, c, d].
ExprAssign	full An assignment expression: a = compute().
ExprAssignOp	full A compound assignment expression: counter += 1.
ExprAsync	full An async block: async { ... }.
ExprAwait	full An await expression: fut.await.
ExprBinary	full or derive A binary operation: a + b, a * b.
ExprBlock	full A blocked scope: { ... }.
ExprBox	full A box expression: box f.
ExprBreak	full A break, with an optional label to break and an optional expression.
ExprCall	full or derive A function call expression: invoke(a, b).
ExprCast	full or derive A cast expression: foo as f64.
ExprClosure	full A closure expression: |a, b| a + b.
ExprContinue	full A continue, with an optional label.
ExprField	full or derive Access of a named struct field (obj.k) or unnamed tuple struct field (obj.0).
ExprForLoop	full A for loop: for pat in expr { ... }.
ExprGroup	full An expression contained within invisible delimiters.
ExprIf	full An if expression with an optional else block: if expr { ... } else { ... }.
ExprIndex	full or derive A square bracketed indexing expression: vector[2].
ExprLet	full A let guard: let Some(x) = opt.
ExprLit	full or derive A literal in place of an expression: 1, "foo".
ExprLoop	full Conditionless loop: loop { ... }.
ExprMacro	full A macro invocation expression: format!("{}", q).
ExprMatch	full A match expression: match n { Some(n) => {}, None => {} }.
ExprMethodCall	full A method call expression: x.foo::<T>(a, b).
ExprParen	full or derive A parenthesized expression: (a + b).
ExprPath	full or derive A path like std::mem::replace possibly containing generic parameters and a qualified self-type.
ExprRange	full A range expression: 1..2, 1.., ..2, 1..=2, ..=2.
ExprReference	full A referencing operation: &a or &mut a.
ExprRepeat	full An array literal constructed from one repeated element: [0u8; N].
ExprReturn	full A return, with an optional value to be returned.
ExprStruct	full A struct literal expression: Point { x: 1, y: 1 }.
ExprTry	full A try-expression: expr?.
ExprTryBlock	full A try block: try { ... }.
ExprTuple	full A tuple expression: (a, b, c, d).
ExprType	full A type ascription expression: foo: f64.
ExprUnary	full or derive A unary operation: !x, *x.
ExprUnsafe	full An unsafe block: unsafe { ... }.
ExprWhile	full A while loop: while expr { ... }.
ExprYield	full A yield expression: yield expr.
Field	full or derive A field of a struct or enum variant.
FieldPat	full A single field in a struct pattern.
FieldValue	full A field-value pair in a struct literal.
FieldsNamed	full or derive Named fields of a struct or struct variant such as Point { x: f64, y: f64 }.
FieldsUnnamed	full or derive Unnamed fields of a tuple struct or tuple variant such as Some(T).
File	full A complete file of Rust source code.
ForeignItemFn	full A foreign function in an extern block.
ForeignItemMacro	full A macro invocation within an extern block.
ForeignItemStatic	full A foreign static item in an extern block: static ext: u8.
ForeignItemType	full A foreign type in an extern block: type void.
Generics	full or derive Lifetimes and type parameters attached to a declaration of a function, enum, trait, etc.
Ident	A word of Rust code, which may be a keyword or legal variable name.
ImplGenerics	(full or derive) and printing Returned by Generics::split_for_impl.
ImplItemConst	full An associated constant within an impl block.
ImplItemMacro	full A macro invocation within an impl block.
ImplItemMethod	full A method within an impl block.
ImplItemType	full An associated type within an impl block.
Index	full or derive The index of an unnamed tuple struct field.
ItemConst	full A constant item: const MAX: u16 = 65535.
ItemEnum	full An enum definition: enum Foo<A, B> { A(A), B(B) }.
ItemExternCrate	full An extern crate item: extern crate serde.
ItemFn	full A free-standing function: fn process(n: usize) -> Result<()> { ... }.
ItemForeignMod	full A block of foreign items: extern "C" { ... }.
ItemImpl	full An impl block providing trait or associated items: impl<A> Trait for Data<A> { ... }.
ItemMacro	full A macro invocation, which includes macro_rules! definitions.
ItemMacro2	full A 2.0-style declarative macro introduced by the macro keyword.
ItemMod	full A module or module declaration: mod m or mod m { ... }.
ItemStatic	full A static item: static BIKE: Shed = Shed(42).
ItemStruct	full A struct definition: struct Foo<A> { x: A }.
ItemTrait	full A trait definition: pub trait Iterator { ... }.
ItemTraitAlias	full A trait alias: pub trait SharableIterator = Iterator + Sync.
ItemType	full A type alias: type Result<T> = std::result::Result<T, MyError>.
ItemUnion	full A union definition: union Foo<A, B> { x: A, y: B }.
ItemUse	full A use declaration: use std::collections::HashMap.
Label	full A lifetime labeling a for, while, or loop.
Lifetime	A Rust lifetime: 'a.
LifetimeDef	full or derive A lifetime definition: 'a: 'b + 'c + 'd.
LitBool	A boolean literal: true or false.
LitByte	A byte literal: b'f'.
LitByteStr	A byte string literal: b"foo".
LitChar	A character literal: 'a'.
LitFloat	A floating point literal: 1f64 or 1.0e10f64.
LitInt	An integer literal: 1 or 1u16.
LitStr	A UTF-8 string literal: "foo".
Local	full A local let binding: let x: u64 = s.parse()?.
Macro	full or derive A macro invocation: println!("{}", mac).
MetaList	full or derive A structured list within an attribute, like derive(Copy, Clone).
MetaNameValue	full or derive A name-value pair within an attribute, like feature = "nightly".
MethodTurbofish	full The ::<> explicit type parameters passed to a method call: parse::<u64>().
ParenthesizedGenericArguments	full or derive Arguments of a function path segment: the (A, B) -> C in Fn(A,B) -> C.
PatBox	full A box pattern: box v.
PatIdent	full A pattern that binds a new variable: ref mut binding @ SUBPATTERN.
PatLit	full A literal pattern: 0.
PatMacro	full A macro in pattern position.
PatOr	full A pattern that matches any one of a set of cases.
PatPath	full A path pattern like Color::Red, optionally qualified with a self-type.
PatRange	full A range pattern: 1..=2.
PatReference	full A reference pattern: &mut var.
PatRest	full The dots in a tuple or slice pattern: [0, 1, ..]
PatSlice	full A dynamically sized slice pattern: [a, b, ref i @ .., y, z].
PatStruct	full A struct or struct variant pattern: Variant { x, y, .. }.
PatTuple	full A tuple pattern: (a, b).
PatTupleStruct	full A tuple struct or tuple variant pattern: Variant(x, y, .., z).
PatType	full A type ascription pattern: foo: f64.
PatWild	full A pattern that matches any value: _.
Path	full or derive A path at which a named item is exported (e.g. std::collections::HashMap).
PathSegment	full or derive A segment of a path together with any path arguments on that segment.
PredicateEq	full or derive An equality predicate in a where clause (unsupported).
PredicateLifetime	full or derive A lifetime predicate in a where clause: 'a: 'b + 'c.
PredicateType	full or derive A type predicate in a where clause: for<'c> Foo<'c>: Trait<'c>.
QSelf	full or derive The explicit Self type in a qualified path: the T in <T as Display>::fmt.
Receiver	full The self argument of an associated method, whether taken by value or by reference.
Signature	full A function signature in a trait or implementation: unsafe fn initialize(&self).
TraitBound	full or derive A trait used as a bound on a type parameter.
TraitItemConst	full An associated constant within the definition of a trait.
TraitItemMacro	full A macro invocation within the definition of a trait.
TraitItemMethod	full A trait method within the definition of a trait.
TraitItemType	full An associated type within the definition of a trait.
Turbofish	(full or derive) and printing Returned by TypeGenerics::as_turbofish.
TypeArray	full or derive A fixed size array type: [T; n].
TypeBareFn	full or derive A bare function type: fn(usize) -> bool.
TypeGenerics	(full or derive) and printing Returned by Generics::split_for_impl.
TypeGroup	full or derive A type contained within invisible delimiters.
TypeImplTrait	full or derive An impl Bound1 + Bound2 + Bound3 type where Bound is a trait or a lifetime.
TypeInfer	full or derive Indication that a type should be inferred by the compiler: _.
TypeMacro	full or derive A macro in the type position.
TypeNever	full or derive The never type: !.
TypeParam	full or derive A generic type parameter: T: Into<String>.
TypeParen	full or derive A parenthesized type equivalent to the inner type.
TypePath	full or derive A path like std::slice::Iter, optionally qualified with a self-type as in <Vec<T> as SomeTrait>::Associated.
TypePtr	full or derive A raw pointer type: *const T or *mut T.
TypeReference	full or derive A reference type: &'a T or &'a mut T.
TypeSlice	full or derive A dynamically sized slice type: [T].
TypeTraitObject	full or derive A trait object type Bound1 + Bound2 + Bound3 where Bound is a trait or a lifetime.
TypeTuple	full or derive A tuple type: (A, B, C, String).
UseGlob	full A glob import in a use item: *.
UseGroup	full A braced group of imports in a use item: {A, B, C}.
UseName	full An identifier imported by a use item: HashMap.
UsePath	full A path prefix of imports in a use item: std::....
UseRename	full An renamed identifier imported by a use item: HashMap as Map.
Variadic	full or derive The variadic argument of a foreign function.
Variant	full or derive An enum variant.
VisCrate	full or derive A crate-level visibility: crate.
VisPublic	full or derive A public visibility level: pub.
VisRestricted	full or derive A visibility level restricted to some path: pub(self) or pub(super) or pub(crate) or pub(in some::module).
WhereClause	full or derive A where clause in a definition: where T: Deserialize<'de>, D: 'static.

Enums

AttrStyle	full or derive Distinguishes between attributes that decorate an item and attributes that are contained within an item.
BinOp	full or derive A binary operator: +, +=, &.
Data	derive The storage of a struct, enum or union data structure.
Expr	full or derive A Rust expression.
Fields	full or derive Data stored within an enum variant or struct.
FnArg	full An argument in a function signature: the n: usize in fn f(n: usize).
ForeignItem	full An item within an extern block.
GenericArgument	full or derive An individual generic argument, like 'a, T, or Item = T.
GenericMethodArgument	full An individual generic argument to a method, like T.
GenericParam	full or derive A generic type parameter, lifetime, or const generic: T: Into<String>, 'a: 'b, const LEN: usize.
ImplItem	full An item within an impl block.
Item	full Things that can appear directly inside of a module or scope.
Lit	A Rust literal such as a string or integer or boolean.
MacroDelimiter	full or derive A grouping token that surrounds a macro body: m!(...) or m!{...} or m![...].
Member	full or derive A struct or tuple struct field accessed in a struct literal or field expression.
Meta	full or derive Content of a compile-time structured attribute.
NestedMeta	full or derive Element of a compile-time attribute list.
Pat	full A pattern in a local binding, function signature, match expression, or various other places.
PathArguments	full or derive Angle bracketed or parenthesized arguments of a path segment.
RangeLimits	full Limit types of a range, inclusive or exclusive.
ReturnType	full or derive Return type of a function signature.
Stmt	full A statement, usually ending in a semicolon.
StrStyle	The style of a string literal, either plain quoted or a raw string like r##"data"##.
TraitBoundModifier	full or derive A modifier on a trait bound, currently only used for the ? in ?Sized.
TraitItem	full An item declaration within the definition of a trait.
Type	full or derive The possible types that a Rust value could have.
TypeParamBound	full or derive A trait or lifetime used as a bound on a type parameter.
UnOp	full or derive A unary operator: *, !, -.
UseTree	full A suffix of an import tree in a use item: Type as Renamed or *.
Visibility	full or derive The visibility level of an item: inherited or pub or pub(restricted).
WherePredicate	full or derive A single predicate in a where clause: T: Deserialize<'de>.

Functions

parse	parsing and proc-macro Parse tokens of source code into the chosen syntax tree node.
parse2	parsing Parse a proc-macro2 token stream into the chosen syntax tree node.
parse_file	parsing and full Parse the content of a file of Rust code.
parse_str	parsing Parse a string of Rust code into the chosen syntax tree node.

Type Definitions

AttributeArgs	full or derive Conventional argument type associated with an invocation of an attribute macro.
Result	The result of a Syn parser.