Expand description
An extractor based low level macro parsing crate that provides high level parsing support through derive macros.
§FromMacro
and Extractor
FromMacro
is the bread and butter of this crate.
FromMacro
provides a from_one
and a from_many
function
that parses a TokenTree
and a TokenStream
respectively.
We can mostly assume from_many
will contain two or more TokenTrees
. If not
that is considered a bug in an extractor.
All implementors of FromMacro
are Extractors
of one TokenTree
. When used on a
TokenStream
iterator directly, they will consume a single TokenTree
and try to parse it using from_one
.
let mut iter = quote!{45, Hello; true false}.into_iter();
let a: i32 = iter.extract()?;
let b: Punct = iter.extract()?;
// Extracts a string value from an ident
let IdentString(c) = iter.extract()?;
// Validates a semicolon
let d: PunctOf<';'> = iter.extract()?;
let e: bool = iter.extract()?;
// Validates a false literal
let f: LitFalse = iter.extract()?;
This is pretty great! As most things can be represented as a single TokenTree
.
// This is a single TokenTree::Group
{
name: "Tom".
age: 45,
children: [
"Tim", "Tam"
],
}
However there are other things one TokenTree
cannot account for.
// This fails because -45 is two tokens
let a: i32 = quote!{-45}.into_iter().extract().unwrap();
Wrapping FromMacro
implementers in other Extractors
allow FromMacro
implementors to parse additional TokenTrees
and
potentially utilize the from_many
method
if more than one TokenTree
is matched.
// Note -45 is two tokens
let mut iter = quote!{-45}.into_iter();
// All extracts everything from a stream
let All(a) = iter.extract()?;
assert_eq!(a, -45i32);
let mut iter = quote!{-45, 21, 9.5,}.into_iter();
// CommaExtractor extracts until a comma or end of stream is found.
let CommaExtractor(a) = iter.extract()?;
let CommaExtractor(b) = iter.extract()?;
let CommaExtractor(c) = iter.extract()?;
// EndOfStream is a unit struct extractor and this asserts iter is empty
let EndOfStream = iter.extract()?;
assert_eq!(a, -45);
assert_eq!(b, 21);
assert_eq!(c, 9.5);
§Derive
We provide derive macro FromMacro
and FromAttrs
that functions similarly to
to serde::Deserialize
. This enables ergonomic
parsing for structs and enums following a specific data format.
FromMacro
parses syntax similar to native rust,
while FromAttrs
parses syntax commonly used in macro attributes.
§Why not serde_tokenstream
?
Since we do not use the serde
data model. Our data model is much more powerful in the macro context.
We are allowed to extract all FromMacro
implementors,
including TokenStream
, Ident
, Group
, etc.
§FromMacro
FromMacro
parses syntax similar to native rust.
Type | from_one | from_many |
---|---|---|
Unit Struct | StructName | – |
Tuple Struct | (tuple, ..) | StructName (tuple, ..) |
Named Struct | {field: value, ..} | StructName {field: value, ..} |
Unit Enum Variant | VariantName | – |
Tuple Enum Variant | – | VariantName (tuple, ..) |
Named Enum Variant | – | VariantName {field: value, ..} |
§Examples
Type | Rust Type | from_one | from_many |
---|---|---|---|
Unit Struct | struct Red; | Red | – |
Tuple Struct | struct Vec2 (i32, i32) | (4, 5) | Vec2 (4, 5) |
Named Struct | struct Vec2 {x: i32, y: i32} | {x: 4, y: 5} | Vec2 {x: 4, y: 5} |
Unit Variant | enum Color {Black, White} | Black | – |
Tuple Variant | enum Animals {Dog(String), Sheep(usize)} | – | Dog ("Rex") |
Named Variant | enum Shapes {Square {x: f32}, Rect {x: f32, y: f32}} | – | Rect {x: 4, y: 5} |
§Use Case
Since we are likely to be parsing configurations in macros,
we supply a Default::default()
value if a field is not found.
You are required to opt out of this with #[macroex(required)]
if
your type does not implement Default
.
#[derive(FromMacro)]
pub struct Person {
pub name: String,
pub age: i32,
pub height: f32,
// This works as long as Gender implements `FromMacro` and `Default`
pub gender: Gender,
// Using Option is idiomatic to handle the default case.
pub hair_color: Option<NumberList<[f32;4]>>,
// We can extract macro based things
pub do_something: Option<TokenStream>,
}
Example macro input:
person! {
name: "Lina",
age: 23,
gender: Female,
hair_color: [0.7, 0.4, 0],
do_something: {
let w = "world";
println!("Hello, {}!", w)
},
}
§Attributes
The FromMacro
macro supports the following attributes:
#[derive(FromMacro)]
// We use the same casing names as serde.
#[macroex(rename_all="SCREAMING-KEBAB-CASE")]
pub struct Animal {
// Errors if not specified.
#[macroex(required)]
pub name: String,
// Evaluate an expression instead of `Default::default()`
#[macroex(default="0.0")]
pub height: f32,
#[macroex(default=r#""dog".to_owned()"#)]
pub species: String,
// Take strings as inputs, and collects them into a vec.
#[macroex(repeat)]
// and rename "nicknames" to "nickname" during parsing.
#[macroex(rename="nickname")]
pub nicknames: Vec<String>,
}
§FromAttrs
FromAttrs
Generates a simple FromMacro
implementation for syntax commonly associated with macro attributes.
This macro is only allowed on named structs and supports 3 basic syntax:
.., name, ..
parses toname: true
, which matches a boolean value..., name = expr, ..
parses toname: expr
.., name(..), ..
parses toname: T{ .. }
Other types like fieldless enums can potentially use FromMacro
to generated compatible FromMacro
implementations to use with this macro.
§Example
We use the same set of attributes as FromMacro
#[derive(FromAttrs)]
#[macroex(rename_all="snake_case")]
pub struct Animal {
#[macroex(required)]
pub name: String,
#[macroex(default="0.0")]
pub height: f32,
#[macroex(default=r#"Ident::new("Dog", Span::call_site())"#)]
pub species: Ident,
#[macroex(repeat, rename="mascot")]
pub mascot_infos: Vec<MascotInfo>,
}
Example attribute:
#[animal(name = "Ferris", species = Crab, mascot(language = "Rust"))]
We can parse either
(name = "Ferris", species = Crab, mascot(language = "Rust"))
with from_one
, or
name = "Ferris", species = Crab, mascot(language = "Rust")
with from_many
, commonly extracted with syn
.
§Macro Chaining and Hygeine
We treat our input as string-like and we will try
to flatten all None
delimited groups encountered during parsing.
Re-exports§
pub use quote;
pub use proc_macro2;
Macros§
- abort_
this - Use
proc_macro_error::abort!
with ourError
. - bail
- Creates and returns a formatted custom
macroex::Error
, similar toanyhow::bail!
. - call_
syntax - Converts an array into a function call in
quote!
. - construct_
syntax - Converts array into a struct constructor in
quote!
. - ident_
validator - Define your own Ident validator.
Structs§
- All
- Extractor that matches everything in the stream and dipatches
from_one
andfrom_many
accordingly - Array
Repeat - Extractor for syntax
[T; N]
- Bracketed
- Extracts a
TokenStream
enclosed in brackets[]
. - Bracketed
Group - Extacts a
Group
enclosed in brackets[]
. - Byte
- Specialized extractor for
u8
that additionally accepts a byte char literal. - Bytes
- Specialized extractor for
Vec<u8>
that additionally accepts a byte string literal. - Colon
Extractor - Extractor that matches until a colon is found.
- Comma
Extractor - Extractor that matches until a comma is found.
- Curly
Braced - Extracts a
TokenStream
enclosed in curly braces{}
. - Curly
Braced Group - Extacts a
Group
enclosed in curly braces{}
. - EndOf
Stream - Special extractor that validates the end of TokenStream.
- Error
- A spanned error. Useful in proc macro error handling.
- Field
- Extracts an extression like
x: 10
,y: -4.0 * f32::PI
,z: Vec2 {x: 1, y: 2}
- HexNumber
- Extracts a Hex number literal and its string representation.
- Ident
String - Extracts a string from an ident.
- Iter
- Extracts a
TokenStream::IntoIter
. - LitFalse
- Unit struct validator for boolean literal
false
. - LitNone
- Unit struct validator for
None
. - LitSome
- Unit struct validator for
Some
. - LitTrue
- Unit struct validator for boolean literal
true
. - Meta
- Matches parsable metalist segment for
FromAttr
. - Multi
Extractor - Extracts multiple items sequentiallly.
- Named
Struct Extractor - Extracts an expression like
Name {..}
- NotBracketed
- Extacts anything that’s not in a
Group
enclosed in brackets[]
. - NotCurly
Braced - Extacts anything that’s not in a
Group
enclosed in curly braces{}
. - NotParenthesized
- Extacts anything that’s not in a
Group
enclosed in parenthesis()
. - Number
- Specialized extractor for
f32
andf64
that additionally accepts integers. - Number
List - Rewrites integer literals in a group into floating point literals.
- OrEnd
OfStream - Extracts an
Option<T>
that may beNone
if at end of stream. - Parenthesized
- Extracts a
TokenStream
enclosed in parenthesis()
. - Parenthesized
Group - Extacts a
Group
enclosed in parenthesis()
. - Punct
Alone - Unit struct validator of a alone punctuation mark.
- Punct
Joint - Unit struct validator of a joint punctuation mark.
- PunctOf
- Unit struct validator of a punctuation mark
- Punctuated
- Extractor that matches until a punctuation mark is found.
- Repeat
- If input
v
is not a Group inDelimiter::Bracket
, rewrite it as[v, v, .., v]
(N times) - Rust
Expr - Extracts an expression according to Rust’s rule.
Ends when encountering
,
,;
,=>
or end of stream. - Semi
Colon Extractor - Extractor that matches until a semicolon is found.
- Spanned
- Records the span from the incoming TokenTree
- Splat
- If input is of the syntax
[v; N]
, rewrite the input into[v, v, .. v,]
(N times) - String
Tokens - Parse a string as
TokenStream
and extract into T. - Stringify
- Convert input tokens to a String using the
to_string
method. - Struct
Extractor - Extracts an expression like
Name {..}
orName (..)
- Tuple
List - Extractor that rewrites
(..)
to[..]
- Tuple
Struct Extractor - Extracts an expression like
Name (..)
- USize
- Unit struct validator of an integer literal.
- Unit
- Unit struct validator for
()
.
Enums§
- Either
- Matches one of many
FromMacro
implementors sequentially. - Either3
- Matches one of many
FromMacro
implementors sequentially. - Either4
- Matches one of many
FromMacro
implementors sequentially. - Either5
- Matches one of many
FromMacro
implementors sequentially. - Either6
- Matches one of many
FromMacro
implementors sequentially. - Either7
- Matches one of many
FromMacro
implementors sequentially. - Either8
- Matches one of many
FromMacro
implementors sequentially. - Either9
- Matches one of many
FromMacro
implementors sequentially. - ExprSep
- Separator of expressions.
- Fallible
- Monadic extractor that converts Err to None.
- OneOr
Many - Matches one of two
FromMacro
implementors by count.
Traits§
- Extractor
- Allow extracting from a TokenStream Iterator.
- From
Macro - A type that can be parsed from a one or more items in a TokenStream.
- Stream
Extract - An extractable stream, like
TokenStream::IntoIter
.
Functions§
- format_
call - Format a fixed sized array into a function call.
- format_
comstructor - Format a fixed sized array into a function call.
- sanitize
- Flattens all none delimited groups and returns the length.
Type Aliases§
- Either
Stream - Either a
TokenTree
or aTokenStream