Derive Macro Tokens

#[derive(Tokens)]
{
    // Attributes available to this derive:
    #[ignored]
    #[token]
}

Derive macro for Tokens

Every token must have a #[token(regex [, process_func])] attribute. process_func is an optional closure that specifies how data should be extracted from the token characters. regex indicates how the token should be matches. The tokens defined first will have higher priority than the later ones.

You can also use any number of #[ignore(regex)] attributes to indicate that some character sequences should be ignored

Panics

Panics if any attributes are invalid