Module doc_implementation 

Implementation approach - how does this work?

[ docs: crate top-level | overall toc, macros | template etc. reference | guide/tutorial ]

You do not need to understand this in order to use derive-deftly.

Also, you should not rely on the details here. They don’t form part of the public interface.

Introduction

It is widely understood that proc macro invocations cannot communicate with each other. (Some people have tried sneaking round the back with disk files etc. but this can break due to incremental and concurrent compilation.)

But, a proc macro can define a macro_rules macro. Then, later proc macros can invoke that macro. The expansion can even invoke further macros. In this way, a proc macro invocation can communicate with subsequent proc macro invocations. (This trick is also used by ambassador.)

There is a further complication. One macro X cannot peer into and dismantle the expansion of another macro Y. (This is necessary for soundness, when macros can generate unsafe.) So we must sometimes define macros that apply other macros (whose name is supplied as an argument) to what is conceptually the first macro’s output.

Implementation approach - reusable template macros

Expansion chaining

When a #[derive(Deftly)] call site invokes multiple templates, we don’t emit separate macro calls for each template. Instead, we generate a macro call for the first template, and pass it the names of subsequent templates. Each template expansion is responsible for invoking the next.

This allows output from the expansion to be threaded through the chain, collecting additional information as we go, and eventually analysed after all the expansions are done. (This is done by puttingderive_deftly_engine as the final entry in the list of macros to chain to.)

Currently, this data collection is used for detecting unused meta attributes.

This does mean that specifying a completely unknown template (for example, one that’s misspelled or not in scope) will prevent error reports from subsequent templates, which is a shame.

Overall input syntax for templates and derive_deftly_engine!

Because of expansion chaining, calls to the engine, and to templates, overlap.

    macro! {
        { DRIVER.. }
        [ 1 0 AOPTIONS.. ] // not present for d_d_dengine expanding ad-hoc
                           // replaced with just "." at end of chain
    // always present, but always immediately ignored:
        ( .... )
    // these parts passed to d_d_engine only, and not in last call:
        { TEMPLATE.. }
        ( CRATE; [TOPTIONS..]
          TEMPLATE_NAME /* omitted if not available */;
          { IMPORTED_DEFINITIONS.. } // from a module (may be omitted)
          .... )
    // always present:
        // after here is simply passed through by templates:
        [
          // usually one or more of these; none at end, or ad-hoc
          CHAIN_TO ( CHAIN_PASS_AFTER_DRIVER .... )
          ..
        ]
        [ACCUM..]
        ....
    }

(We use the notation .... for room we are leaving for future expansion; these are currently empty.)

The engine also accepts a nonoverlapping syntax for handling template definitions which use modules - see below.

1. define_derive_deftly! macro for defining a reuseable template

Implemented in define.rs::define_derive_deftly_func_macro.

When used like this

    define_derive_deftly! {
        MyMacro TOPTIONS..:
        TEMPLATE..
    }

Expands to

    macro_rules! derive_deftly_template_Template { {
        { $($driver:tt)* }
        [ $($aoptions:tt)* ]
        ( $($future:tt)* )
        $($tpassthrough:tt)*
    } => {
        derive_deftly_engine! {
            { $($driver)* }
            [ $(aoptions)* ]
            ( )
            { TEMPLATE.. }
            ( $crate; [TOPTIONS..] Template;
              { IMPORTED_DEFINITIONS.. } // only if via modules (see below)
              )
            $($tpassthrough)*
        }
    } }

Except, every $ in the TEMPLATE is replaced with $orig_dollar. This is because a macro_rules! template is not capable of generating a literal in the expansion $. (With the still-unstable decl_macro feature, $$ does this.) macro_rules! simply passes through $orig_dollar (as it does with all unrecognised variables), and the template expansion engine treats $orig_dollar as just a single $.

(The extra ( ) parts after the driver and template include space for future expansion.)

2. #[derive_deftly(Template)], implemented in #[derive(Deftly)]

Template use is implemented in derive.rs::derive_deftly.

This

    #[derive(Deftly)]
    #[derive_deftly(Template[AOPTIONS..], Template2)]
    pub struct StructName { .. }

Generates:

    derive_deftly_template_Template! {
        { #[derive_deftly(..)] struct StructName { .. } }
        [ 1 0 AOPTIONS ]
        ( )
        [
          derive_deftly_template_Template2 ( [1 0 AOPTIONS2] () )
          derive_deftly_engine ( . () )
        ]
        [] // (nothing accumulated yet)
    }

3. Actual expansion

The first call to derive_deftly_template_Template! is expanded according to the macro_rules! definition, resulting in a call to derive_deftly_engine:

    derive_deftly_engine! {
        { #[derive_deftly(..)] pub struct StructName { .. } }
        [ 1 0 AOPTIONS ]
        ( )
        { TEMPLATE.. }
        ( $crate; [TOPTIONS..] Template; )
        [
          derive_deftly_template_Template2 ( [1 0 AOPTIONS2] () )
          derive_deftly_engine ( . () )
        ]
        [] // ACCUM
    }

Implemented in engine.rs::derive_deftly_engine_func_macro, this performs the actual template expansion.

4. Chaining to the next macro

derive_deftly_engine! then invokes the next template.

In the above example, it outputs:

    derive_deftly_template_Template2! {
        { #[derive_deftly(..)] pub struct StructName { .. } }
        [ 1 0 AOPTIONS2 ]
        ( ) // threaded through from the call site
        [ derive_deftly_engine ( . () ) ]
        [_name Template _meta_used [..] _meta_recog [..]] // ACCUM
    }

ACCUM accumulates information from successive expansions, and is actually parsed only in the next step.

Used meta attribute syntax

_meta_used introduces the meta attributes from the driver, which were used by this template, in the following syntax:

  ::VARIANT // entries until next :: are for this variant
  .field    // entries until next :: or . are for this field
  ( // each (..) corresponds to one #[deftly()], and matches its tree
    somemeta(
      value_used =, // value was used
      value_used +, // value was used but in context with a default
      bool_tested ?, // boolean was tested
      , // skip a node (subtree) not used at all
      .. // trailing skipped notes are omitted
    )
  ) // empty trailing groups are omitted

So for example this

struct S {
  #[adhoc(foo(bar, baz))]
  #[adhoc(wombat, zoo = "Berlin")]
  f: T,
}

might correspond to this

_meta [
    // nothing (don't need field name even)
    .f () (, zoo?)
    .f () (wombat?, zoo=)
    .f (foo=(bar?, baz?)) (wombat?, zoo?)
]

This representation helps minimise the amount of text which needs to be passed through all the macro chains, when only a few attributes are used by each template, while still being unambiguous and detecting desynchronisation.

Instead of a [..] list, _meta_used can also be *, meaning that no checking should be done.

Recognised meta attribute syntax

_meta_recog introduces the meta attributes which might conceivably be recognised by this template. This is determined statically, by scanning the template.

The information is used only if there are unused attributes, to assist with producing good error messages.

_meta_recog takes a [ ] list containing items like fmeta(foo(bar)), or fmeta(foo(bar))?` when recognised only as a boolean, or fmeta(foo(bar))+` when only used as a value (without any boolean tests).

5. Reporting unused meta attributes

At the end of the list of chained templates, is (the driver’s version of) derive_deftly_engine!. So after all the templates have been processed, instead of invoking the next template, we return directly to the engine:

    derive_deftly_engine! {
        { #[derive_deftly(..)] pub struct StructName { .. } }
        .
        ( )
        [] // end of the chain
        [_name Template _meta_used [..] _meta_recog [..] _name Template2 ..]
    }

The accumulated parts are all of the form: KEYWORD DATA; _KEYWORD DATA for a part which can be safely ignored, if unrecognised. DATA is a single tt. Each template’s parts start with a _name part. There can also be error parts which appear when the template couldn’t be parsed (which is used to suppress “unrecognised attribute” errors).

The actually supplied attributes are compared with the union of the used attributes, and errors are reported for unused ones.

Implementation approach - ad-hoc macro applications

1. #[derive(Deftly)] feature for saving struct definitions

Also implemented in derive.rs::derive_deftly.

When applied to (e.g.) pub struct StructName, with #[derive_deftly_adhoc] specified generates this

    macro_rules! derive_deftly_driver_StructName { {
        { $($template:tt)* }
        { ($orig_dollar:tt) $(future:tt)* }
        $($dpassthrough:tt)*
     } => {
        derive_deftly_engine!{
            { pub struct StructName { /* original struct definition */ } }
            /* no AOPTIONS since there's no derive() application */
            ( )
            { $($template)* }
            $($dpassthrough)*
        }
    } }

(Again, the extra { } parts after the driver and template include space for future expansion.)

In the pub struct part every $ is replaced with $orig_dollar, to use the $ passed in at the invocation site. (This is only relevant if the driver contains $ somehow, for example in helper attributes.)

2. derive_deftly_adhoc! function-like proc macro for applying to a template

Implemented in adhoc.rs::derive_deftly_adhoc.

When applied like this

    derive_deftly_adhoc!{
       StructName TOPTIONS..:
       TEMPLATE..
    }

Expands to

    derive_deftly_driver_StructName! {
       { TEMPLATE.. }
       { ($) }
       ( crate; [TOPTIONS..] /*no template name*/; )
       []
       [_meta_used *]
    }

The literal $ is there to work around a limitation in macro_rules!, see above.

3. Function-like proc macro to do the actual expansion

The result of expanding the above is this:

    derive_deftly_engine!{
        { pub struct StructName { /* original struct definition */ } }
        ( )
        { TEMPLATE.. }
        ( crate; [TOPTIONS..] /*no template name*/; )
        []
        [_meta_used *]
    }

derive_deftly_engine parses pub struct StructName, and implements a bespoke template expander, whose template syntax resembles the expansion syntax from macro_rules.

crate is just used as the expansion for ${crate}. (For an ad-hoc template, the local crate is correct.)

Implementation approach - deftly definition modules

Overall input syntax for modules and derive_deftly_engine!

    macro! {
	    via_modules [
            {....} // used by `macro`
		    NEXT_MACRO {....} // used by `NEXT_MACRO`
	        ..
			define_derive_deftly {....} DEFWHAT ....
	    ]
        { EARLIER_PREFIX.. }
        { MAIN_PASSTHROUGH.. }
        ( $ EXTRA_PASSTHROUGH.... )
        ....
    }

Loosely, the semantics of derive_deftly_module_MODULE are: define a thing, which depends on MODULE’s definitions.

Each derive_deftly_module_MODULE macro prepends its own definitions to EARLIER_PREFIX, leaving MAIN_PASSTHROUGH unchanged, and then calls the next macro in turn.

Eventually derive_deftly_engine is reached, with NEXT_MACROS empty, and looks at [ DEFWHAT .... ] to know what to do.

When a template uses multiple modules, the module macros are listed in [ ] in reverse order, because the last macro in the list runs last, prepending, so that its definitions end up first in the output. Engine comes last because it runs on the “outside”, after all the prepending is done. The reverse ordering is not strictly necessary - we could define the macros to append and list in forwards order; it’s done this way for future compatibility with possible other uses of the module system.

EARLIER_PREFIX has already been dollar-escaped, using $ orig_dollar. MAIN_PASSTHROUGH has not.

Module macro

    define_derive_deftly_module! {
        DOCS
        [export] MODULE:

        M_DEFINES
    }

Expands to:

    DOCS
    #[macro_export]
    macro_rules! derive_deftly_module_MODULE { {
        via_modules [
            { $($our_opts:tt)* }
		    $next_macro:path
	        { $($next_opts:tt)* }
			$(rest:tt)*
		]
        { $($predefs:tt)* }
        { $($main:tt)* }
        ( $($extra:tt)* )
        $($ignored:tt)*
    } => {
      $next_macro! {
        via_modules [ { $($next_opts)* } $($rest)* ]
        { M_DEFINES $($predefs)* }
        { $($main)* }
        ( $($extra)* )
      }
    } }

In M_DEFINES, dollars have to be turned into $orig_dollar.

Definition of a module-using template

    define_derive_deftly! {
        use M1;
        use M2;
        DOCS MyMacro TOPTIONS..: TEMPLATE..
    }

Expands to:

    derive_deftly_module_M2! {
        via_modules [
		    {}
			derive_deftly_module_M1 {}
			derive_deftly_engine {} define
        ]
        { }
        { DOCS MyMacro TOPTIONS..: TEMPLATE.. }
        ( )
    }

Hence:

    derive_deftly_module_M1! {
        via_modules [
		    {}
			derive_deftly_engine {} define
        ]
        { M2_DEFINES }
        { DOCS MyMacro TOPTIONS..: TEMPLATE.. }
        ( )
    }

Hence:

    derive_deftly_engine! {
        via_modules [
		    {} define
	    ]
        { M1_DEFINES M2_DEFINES }
        { DOCS MyMacro TOPTIONS..: TEMPLATE.. }
        ( )
    }

Which is then handled as if it were an invocation of define_derive_deftly! without any use statements.

The resulting template macro will additionally embody { M1_DEFINES M2_DEFINES } as { IMPORTED_DEFINITIONS.. }.

Definition of a module-using module

These are resolved early, so that they are syntax-checked (and the names of the imported modules resolved) at the definition site:

    define_derive_deftly_module! {
        MODULE_DOCS
        [export] MODULE:

        use P1;
        use P2;

        M_DEFINES
    }

Expands to:

    derive_deftly_module_P2! {
        via_modules [
		    {}
			derive_deftly_module_P1 {}
			derive_deftly_engine {} defmod
		]
        { M_DEFINES }
        { MODULE_DOCS [export] MODULE: }
        ( )
    }

Hence:

    derive_deftly_module_P1! {
        via_modules [
		    {}
			derive_deftly_engine {} defmod
		]
        { P2_DEFINES M_DEFINES }
        { MODULE_DOCS [export] MODULE: }
        ( )
    }

Hence:

    derive_deftly_engine {
        via_modules [
		    {} defmod
        ]
        { P1_DEFINES P2_DEFINES M_DEFINES }
        { MODULE_DOCS [export] MODULE: }
        ( )
    }

Which is treated as:

    define_derive_deftly_module! {
        MODULE_DOCS [export] MODULE:
        P1_DEFINES P2_DEFINES M_DEFINES
    }