Expand description
Syn is a parsing library for parsing a stream of Rust tokens into a syntax tree of Rust source code.
Currently this library is geared toward use in Rust procedural macros, but contains some APIs that may be useful more generally.
-
Data structures — Syn provides a complete syntax tree that can represent any valid Rust source code. The syntax tree is rooted at
syn::File
which represents a full source file, but there are other entry points that may be useful to procedural macros includingsyn::Item
,syn::Expr
andsyn::Type
. -
Derives — Of particular interest to derive macros is
syn::DeriveInput
which is any of the three legal input items to a derive macro. An example below shows using this type in a library that can derive implementations of a user-defined trait. -
Parsing — Parsing in Syn is built around parser functions with the signature
fn(ParseStream) -> Result<T>
. Every syntax tree node defined by Syn is individually parsable and may be used as a building block for custom syntaxes, or you may dream up your own brand new syntax without involving any of our syntax tree types. -
Location information — Every token parsed by Syn is associated with a
Span
that tracks line and column information back to the source of that token. These spans allow a procedural macro to display detailed error messages pointing to all the right places in the user’s code. There is an example of this below. -
Feature flags — Functionality is aggressively feature gated so your procedural macros enable only what they need, and do not pay in compile time for all the rest.
§Example of a derive macro
The canonical derive macro using Syn looks like this. We write an ordinary
Rust function tagged with a proc_macro_derive
attribute and the name of
the trait we are deriving. Any time that derive appears in the user’s code,
the Rust compiler passes their data structure as tokens into our macro. We
get to execute arbitrary Rust code to figure out what to do with those
tokens, then hand some tokens back to the compiler to compile into the
user’s crate.
[dependencies]
syn = "2.0"
quote = "1.0"
[lib]
proc-macro = true
use proc_macro::TokenStream;
use quote::quote;
use syn::{parse_macro_input, DeriveInput};
#[proc_macro_derive(MyMacro)]
pub fn my_macro(input: TokenStream) -> TokenStream {
// Parse the input tokens into a syntax tree
let input = parse_macro_input!(input as DeriveInput);
// Build the output, possibly using quasi-quotation
let expanded = quote! {
// ...
};
// Hand the output tokens back to the compiler
TokenStream::from(expanded)
}
The heapsize
example directory shows a complete working implementation
of a derive macro. The example derives a HeapSize
trait which computes an
estimate of the amount of heap memory owned by a value.
pub trait HeapSize {
/// Total number of bytes of heap memory owned by `self`.
fn heap_size_of_children(&self) -> usize;
}
The derive macro allows users to write #[derive(HeapSize)]
on data
structures in their program.
#[derive(HeapSize)]
struct Demo<'a, T: ?Sized> {
a: Box<T>,
b: u8,
c: &'a str,
d: String,
}
§Spans and error reporting
The token-based procedural macro API provides great control over where the
compiler’s error messages are displayed in user code. Consider the error the
user sees if one of their field types does not implement HeapSize
.
#[derive(HeapSize)]
struct Broken {
ok: String,
bad: std::thread::Thread,
}
By tracking span information all the way through the expansion of a
procedural macro as shown in the heapsize
example, token-based macros in
Syn are able to trigger errors that directly pinpoint the source of the
problem.
error[E0277]: the trait bound `std::thread::Thread: HeapSize` is not satisfied
--> src/main.rs:7:5
|
7 | bad: std::thread::Thread,
| ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `HeapSize` is not implemented for `Thread`
§Parsing a custom syntax
The lazy-static
example directory shows the implementation of a
functionlike!(...)
procedural macro in which the input tokens are parsed
using Syn’s parsing API.
The example reimplements the popular lazy_static
crate from crates.io as a
procedural macro.
lazy_static! {
static ref USERNAME: Regex = Regex::new("^[a-z0-9_-]{3,16}$").unwrap();
}
The implementation shows how to trigger custom warnings and error messages on the macro input.
warning: come on, pick a more creative name
--> src/main.rs:10:16
|
10 | static ref FOO: String = "lazy_static".to_owned();
| ^^^
§Testing
When testing macros, we often care not just that the macro can be used
successfully but also that when the macro is provided with invalid input it
produces maximally helpful error messages. Consider using the trybuild
crate to write tests for errors that are emitted by your macro or errors
detected by the Rust compiler in the expanded code following misuse of the
macro. Such tests help avoid regressions from later refactors that
mistakenly make an error no longer trigger or be less helpful than it used
to be.
§Debugging
When developing a procedural macro it can be helpful to look at what the
generated code looks like. Use cargo rustc -- -Zunstable-options --pretty=expanded
or the cargo expand
subcommand.
To show the expanded code for some crate that uses your procedural macro,
run cargo expand
from that crate. To show the expanded code for one of
your own test cases, run cargo expand --test the_test_case
where the last
argument is the name of the test file without the .rs
extension.
This write-up by Brandon W Maister discusses debugging in more detail: Debugging Rust’s new Custom Derive system.
§Optional features
Syn puts a lot of functionality behind optional features in order to optimize compile time for the most common use cases. The following features are available.
derive
(enabled by default) — Data structures for representing the possible input to a derive macro, including structs and enums and types.full
— Data structures for representing the syntax tree of all valid Rust source code, including items and expressions.parsing
(enabled by default) — Ability to parse input tokens into a syntax tree node of a chosen type.printing
(enabled by default) — Ability to print a syntax tree node as tokens of Rust source code.visit
— Trait for traversing a syntax tree.visit-mut
— Trait for traversing and mutating in place a syntax tree.fold
— Trait for transforming an owned syntax tree.clone-impls
(enabled by default) — Clone impls for all syntax tree types.extra-traits
— Debug, Eq, PartialEq, Hash impls for all syntax tree types.proc-macro
(enabled by default) — Runtime dependency on the dynamic library libproc_macro from rustc toolchain.
Modules§
- buffer
parsing
A stably addressed token buffer supporting efficient traversal based on a cheaply copyable cursor. - ext
parsing
Extension traits to provide parsing methods on foreign types. - fold
fold
Syntax tree traversal to transform the nodes of an owned syntax tree. - meta
parsing
and (full
orderive
)Facility for interpreting structured content inside of anAttribute
. - parse
parsing
Parsing interface for parsing a token stream into a syntax tree node. - A punctuated sequence of syntax tree nodes separated by punctuation.
- spanned
parsing
andprinting
A trait that can provide theSpan
of the complete contents of a syntax tree node. - Tokens representing Rust punctuation, keywords, and delimiters.
- visit
visit
Syntax tree traversal to walk a shared borrow of a syntax tree. - visit_
mut visit-mut
Syntax tree traversal to mutate an exclusive borrow of a syntax tree in place.
Macros§
- A type-macro that expands to the name of the Rust type representation of a given token.
- braced
parsing
Parse a set of curly braces and expose their content to subsequent parsers. - bracketed
parsing
Parse a set of square brackets and expose their content to subsequent parsers. - Define a type that supports parsing and printing a given identifier as if it were a keyword.
- Define a type that supports parsing and printing a multi-character symbol as if it were a punctuation token.
- parenthesized
parsing
Parse a set of parentheses and expose their content to subsequent parsers. - parse_
macro_ input parsing
andproc-macro
Parse the input TokenStream of a macro, triggering a compile error if the tokens fail to parse. - parse_
quote parsing
andprinting
Quasi-quotation macro that accepts input like thequote!
macro but uses type inference to figure out a return type for those tokens. - parse_
quote_ spanned parsing
andprinting
This macro isparse_quote!
+quote_spanned!
.
Structs§
- Abi
full
orderive
The binary interface of a function:extern "C"
. - Angle
Bracketed Generic Arguments full
orderive
Angle bracketed arguments of a path segment: the<K, V>
inHashMap<K, V>
. - Arm
full
One arm of amatch
expression:0..=10 => { return true; }
. - Assoc
Const full
orderive
An equality constraint on an associated constant: thePANIC = false
inTrait<PANIC = false>
. - Assoc
Type full
orderive
A binding (equality constraint) on an associated type: theItem = u8
inIterator<Item = u8>
. - Attribute
full
orderive
An attribute, like#[repr(transparent)]
. - Bare
FnArg full
orderive
An argument in a function type: theusize
infn(usize) -> bool
. - Bare
Variadic full
orderive
The variadic argument of a function pointer likefn(usize, ...)
. - Block
full
A braced block containing Rust statements. - Bound
Lifetimes full
orderive
A set of bound lifetimes:for<'a, 'b, 'c>
. - Const
Param full
orderive
A const generic parameter:const LENGTH: usize
. - Constraint
full
orderive
An associated type bound:Iterator<Item: Display>
. - Data
Enum derive
An enum input to aproc_macro_derive
macro. - Data
Struct derive
A struct input to aproc_macro_derive
macro. - Data
Union derive
An untagged union input to aproc_macro_derive
macro. - Derive
Input derive
Data structure sent to aproc_macro_derive
macro. - Error returned when a Syn parser cannot parse the input tokens.
- Expr
Array full
A slice literal expression:[a, b, c, d]
. - Expr
Assign full
An assignment expression:a = compute()
. - Expr
Async full
An async block:async { ... }
. - Expr
Await full
An await expression:fut.await
. - Expr
Binary full
orderive
A binary operation:a + b
,a += b
. - Expr
Block full
A blocked scope:{ ... }
. - Expr
Break full
Abreak
, with an optional label to break and an optional expression. - Expr
Call full
orderive
A function call expression:invoke(a, b)
. - Expr
Cast full
orderive
A cast expression:foo as f64
. - Expr
Closure full
A closure expression:|a, b| a + b
. - Expr
Const full
A const block:const { ... }
. - Expr
Continue full
Acontinue
, with an optional label. - Expr
Field full
orderive
Access of a named struct field (obj.k
) or unnamed tuple struct field (obj.0
). - Expr
ForLoop full
A for loop:for pat in expr { ... }
. - Expr
Group full
An expression contained within invisible delimiters. - ExprIf
full
Anif
expression with an optionalelse
block:if expr { ... } else { ... }
. - Expr
Index full
orderive
A square bracketed indexing expression:vector[2]
. - Expr
Infer full
The inferred value of a const generic argument, denoted_
. - ExprLet
full
Alet
guard:let Some(x) = opt
. - ExprLit
full
orderive
A literal in place of an expression:1
,"foo"
. - Expr
Loop full
Conditionless loop:loop { ... }
. - Expr
Macro full
orderive
A macro invocation expression:format!("{}", q)
. - Expr
Match full
Amatch
expression:match n { Some(n) => {}, None => {} }
. - Expr
Method Call full
orderive
A method call expression:x.foo::<T>(a, b)
. - Expr
Paren full
orderive
A parenthesized expression:(a + b)
. - Expr
Path full
orderive
A path likestd::mem::replace
possibly containing generic parameters and a qualified self-type. - Expr
Range full
A range expression:1..2
,1..
,..2
,1..=2
,..=2
. - Expr
RawAddr full
Address-of operation:&raw const place
or&raw mut place
. - Expr
Reference full
orderive
A referencing operation:&a
or&mut a
. - Expr
Repeat full
An array literal constructed from one repeated element:[0u8; N]
. - Expr
Return full
Areturn
, with an optional value to be returned. - Expr
Struct full
orderive
A struct literal expression:Point { x: 1, y: 1 }
. - ExprTry
full
A try-expression:expr?
. - Expr
TryBlock full
A try block:try { ... }
. - Expr
Tuple full
A tuple expression:(a, b, c, d)
. - Expr
Unary full
orderive
A unary operation:!x
,*x
. - Expr
Unsafe full
An unsafe block:unsafe { ... }
. - Expr
While full
A while loop:while expr { ... }
. - Expr
Yield full
A yield expression:yield expr
. - Field
full
orderive
A field of a struct or enum variant. - Field
Pat full
A single field in a struct pattern. - Field
Value full
orderive
A field-value pair in a struct literal. - Fields
Named full
orderive
Named fields of a struct or struct variant such asPoint { x: f64, y: f64 }
. - Fields
Unnamed full
orderive
Unnamed fields of a tuple struct or tuple variant such asSome(T)
. - File
full
A complete file of Rust source code. - Foreign
Item Fn full
A foreign function in anextern
block. - Foreign
Item Macro full
A macro invocation within an extern block. - A foreign static item in an
extern
block:static ext: u8
. - Foreign
Item Type full
A foreign type in anextern
block:type void
. - Generics
full
orderive
Lifetimes and type parameters attached to a declaration of a function, enum, trait, etc. - A word of Rust code, which may be a keyword or legal variable name.
- Impl
Generics ( full
orderive
) andprinting
Returned byGenerics::split_for_impl
. - Impl
Item Const full
An associated constant within an impl block. - Impl
Item Fn full
An associated function within an impl block. - Impl
Item Macro full
A macro invocation within an impl block. - Impl
Item Type full
An associated type within an impl block. - Index
full
orderive
The index of an unnamed tuple struct field. - Item
Const full
A constant item:const MAX: u16 = 65535
. - Item
Enum full
An enum definition:enum Foo<A, B> { A(A), B(B) }
. - Item
Extern Crate full
Anextern crate
item:extern crate serde
. - ItemFn
full
A free-standing function:fn process(n: usize) -> Result<()> { ... }
. - Item
Foreign Mod full
A block of foreign items:extern "C" { ... }
. - Item
Impl full
An impl block providing trait or associated items:impl<A> Trait for Data<A> { ... }
. - Item
Macro full
A macro invocation, which includesmacro_rules!
definitions. - ItemMod
full
A module or module declaration:mod m
ormod m { ... }
. - Item
Static full
A static item:static BIKE: Shed = Shed(42)
. - Item
Struct full
A struct definition:struct Foo<A> { x: A }
. - Item
Trait full
A trait definition:pub trait Iterator { ... }
. - Item
Trait Alias full
A trait alias:pub trait SharableIterator = Iterator + Sync
. - Item
Type full
A type alias:type Result<T> = std::result::Result<T, MyError>
. - Item
Union full
A union definition:union Foo<A, B> { x: A, y: B }
. - ItemUse
full
A use declaration:use std::collections::HashMap
. - Label
full
A lifetime labeling afor
,while
, orloop
. - A Rust lifetime:
'a
. - Lifetime
Param full
orderive
A lifetime definition:'a: 'b + 'c + 'd
. - A boolean literal:
true
orfalse
. - A byte literal:
b'f'
. - A byte string literal:
b"foo"
. - A nul-terminated C-string literal:
c"foo"
. - A character literal:
'a'
. - A floating point literal:
1f64
or1.0e10f64
. - An integer literal:
1
or1u16
. - A UTF-8 string literal:
"foo"
. - Local
full
A locallet
binding:let x: u64 = s.parse()?
. - Local
Init full
The expression assigned in a locallet
binding, including optional divergingelse
block. - Macro
full
orderive
A macro invocation:println!("{}", mac)
. - Meta
List full
orderive
A structured list within an attribute, likederive(Copy, Clone)
. - Meta
Name Value full
orderive
A name-value pair within an attribute, likefeature = "nightly"
. - Parenthesized
Generic Arguments full
orderive
Arguments of a function path segment: the(A, B) -> C
inFn(A,B) -> C
. - PatConst
full
A const block:const { ... }
. - PatIdent
full
A pattern that binds a new variable:ref mut binding @ SUBPATTERN
. - PatLit
full
A literal in place of an expression:1
,"foo"
. - PatMacro
full
A macro invocation expression:format!("{}", q)
. - PatOr
full
A pattern that matches any one of a set of cases. - PatParen
full
A parenthesized pattern:(A | B)
. - PatPath
full
A path likestd::mem::replace
possibly containing generic parameters and a qualified self-type. - PatRange
full
A range expression:1..2
,1..
,..2
,1..=2
,..=2
. - PatReference
full
A reference pattern:&mut var
. - PatRest
full
The dots in a tuple or slice pattern:[0, 1, ..]
. - PatSlice
full
A dynamically sized slice pattern:[a, b, ref i @ .., y, z]
. - PatStruct
full
A struct or struct variant pattern:Variant { x, y, .. }
. - PatTuple
full
A tuple pattern:(a, b)
. - PatTuple
Struct full
A tuple struct or tuple variant pattern:Variant(x, y, .., z)
. - PatType
full
A type ascription pattern:foo: f64
. - PatWild
full
A pattern that matches any value:_
. - Path
full
orderive
A path at which a named item is exported (e.g.std::collections::HashMap
). - Path
Segment full
orderive
A segment of a path together with any path arguments on that segment. - Precise
Capture full
Precise capturing bound: the ‘use<…>’ inimpl Trait + use<'a, T>
. - Predicate
Lifetime full
orderive
A lifetime predicate in awhere
clause:'a: 'b + 'c
. - Predicate
Type full
orderive
A type predicate in awhere
clause:for<'c> Foo<'c>: Trait<'c>
. - QSelf
full
orderive
The explicit Self type in a qualified path: theT
in<T as Display>::fmt
. - Receiver
full
Theself
argument of an associated method. - Signature
full
A function signature in a trait or implementation:unsafe fn initialize(&self)
. - Stmt
Macro full
A macro invocation in statement position. - Trait
Bound full
orderive
A trait used as a bound on a type parameter. - Trait
Item Const full
An associated constant within the definition of a trait. - Trait
Item Fn full
An associated function within the definition of a trait. - Trait
Item Macro full
A macro invocation within the definition of a trait. - Trait
Item Type full
An associated type within the definition of a trait. - Turbofish
( full
orderive
) andprinting
Returned byTypeGenerics::as_turbofish
. - Type
Array full
orderive
A fixed size array type:[T; n]
. - Type
Bare Fn full
orderive
A bare function type:fn(usize) -> bool
. - Type
Generics ( full
orderive
) andprinting
Returned byGenerics::split_for_impl
. - Type
Group full
orderive
A type contained within invisible delimiters. - Type
Impl Trait full
orderive
Animpl Bound1 + Bound2 + Bound3
type whereBound
is a trait or a lifetime. - Type
Infer full
orderive
Indication that a type should be inferred by the compiler:_
. - Type
Macro full
orderive
A macro in the type position. - Type
Never full
orderive
The never type:!
. - Type
Param full
orderive
A generic type parameter:T: Into<String>
. - Type
Paren full
orderive
A parenthesized type equivalent to the inner type. - Type
Path full
orderive
A path likestd::slice::Iter
, optionally qualified with a self-type as in<Vec<T> as SomeTrait>::Associated
. - TypePtr
full
orderive
A raw pointer type:*const T
or*mut T
. - Type
Reference full
orderive
A reference type:&'a T
or&'a mut T
. - Type
Slice full
orderive
A dynamically sized slice type:[T]
. - Type
Trait Object full
orderive
A trait object typedyn Bound1 + Bound2 + Bound3
whereBound
is a trait or a lifetime. - Type
Tuple full
orderive
A tuple type:(A, B, C, String)
. - UseGlob
full
A glob import in ause
item:*
. - UseGroup
full
A braced group of imports in ause
item:{A, B, C}
. - UseName
full
An identifier imported by ause
item:HashMap
. - UsePath
full
A path prefix of imports in ause
item:std::...
. - UseRename
full
An renamed identifier imported by ause
item:HashMap as Map
. - Variadic
full
The variadic argument of a foreign function. - Variant
full
orderive
An enum variant. - VisRestricted
full
orderive
A visibility level restricted to some path:pub(self)
orpub(super)
orpub(crate)
orpub(in some::module)
. - Where
Clause full
orderive
Awhere
clause in a definition:where T: Deserialize<'de>, D: 'static
.
Enums§
- Attr
Style full
orderive
Distinguishes between attributes that decorate an item and attributes that are contained within an item. - BinOp
full
orderive
A binary operator:+
,+=
,&
. - Captured
Param full
Single parameter in a precise capturing bound. - Data
derive
The storage of a struct, enum or union data structure. - Expr
full
orderive
A Rust expression. - Field
Mutability full
orderive
Unused, but reserved for RFC 3323 restrictions. - Fields
full
orderive
Data stored within an enum variant or struct. - FnArg
full
An argument in a function signature: then: usize
infn f(n: usize)
. - Foreign
Item full
An item within anextern
block. - Generic
Argument full
orderive
An individual generic argument, like'a
,T
, orItem = T
. - Generic
Param full
orderive
A generic type parameter, lifetime, or const generic:T: Into<String>
,'a: 'b
,const LEN: usize
. - Impl
Item full
An item within an impl block. - Impl
Restriction full
Unused, but reserved for RFC 3323 restrictions. - Item
full
Things that can appear directly inside of a module or scope. - A Rust literal such as a string or integer or boolean.
- Macro
Delimiter full
orderive
A grouping token that surrounds a macro body:m!(...)
orm!{...}
orm![...]
. - Member
full
orderive
A struct or tuple struct field accessed in a struct literal or field expression. - Meta
full
orderive
Content of a compile-time structured attribute. - Pat
full
A pattern in a local binding, function signature, match expression, or various other places. - Path
Arguments full
orderive
Angle bracketed or parenthesized arguments of a path segment. - Mutability of a raw pointer (
*const T
,*mut T
), in which non-mutable isn’t the implicit default. - Range
Limits full
Limit types of a range, inclusive or exclusive. - Return
Type full
orderive
Return type of a function signature. - Static
Mutability full
The mutability of anItem::Static
orForeignItem::Static
. - Stmt
full
A statement, usually ending in a semicolon. - Trait
Bound Modifier full
orderive
A modifier on a trait bound, currently only used for the?
in?Sized
. - Trait
Item full
An item declaration within the definition of a trait. - Type
full
orderive
The possible types that a Rust value could have. - Type
Param Bound full
orderive
A trait or lifetime used as a bound on a type parameter. - UnOp
full
orderive
A unary operator:*
,!
,-
. - UseTree
full
A suffix of an import tree in ause
item:Type as Renamed
or*
. - Visibility
full
orderive
The visibility level of an item: inherited orpub
orpub(restricted)
. - Where
Predicate full
orderive
A single predicate in awhere
clause:T: Deserialize<'de>
.
Functions§
- parse
parsing
andproc-macro
Parse tokens of source code into the chosen syntax tree node. - parse2
parsing
Parse a proc-macro2 token stream into the chosen syntax tree node. - parse_
file parsing
andfull
Parse the content of a file of Rust code. - parse_
str parsing
Parse a string of Rust code into the chosen syntax tree node.
Type Aliases§
- The result of a Syn parser.