Crate token_stream_flatten

Flattening iterator adaptor for token streams.

Allows processing rust source as a sequence of primitve tokens.

Usage

The adaptor FlattenRec can be obtained by the flatten_rec method, provided by the trait IntoFlattenRec, implemented for proc-macro2's IntoIter. This is convenient for method chaining.

Alternatively, FlattenRec's From<IntoIter> implementation can be used directly.

let stream = r#"if n > 1 { foo[n - 2] } else { 11 }"#
    .parse::<proc_macro2::TokenStream>()
    .unwrap();
let flat = stream.into_iter().flatten_rec().collect::<Vec<_>>();

assert_eq!(
    flat.iter().map(|token| token.to_string()).collect::<Vec<_>>(),
    vec!["if", "n", ">", "1", "{", "foo", "[", "n", "-", "2", "]", "}", "else", "{", "11", "}"],
);

assert_eq!(flat[2].span().start(), proc_macro2::LineColumn { line: 1, column: 5 });
assert_eq!(flat[2].span().end(), proc_macro2::LineColumn { line: 1, column: 6 });
if let Token::Punct(punct) = &flat[2] {
    assert_eq!(punct.as_char(), '>');
} else {
    panic!("expected punct");
}

assert_eq!(flat[10].span().start(), proc_macro2::LineColumn { line: 1, column: 20 });
assert_eq!(flat[10].span().end(), proc_macro2::LineColumn { line: 1, column: 21 });
if let Token::Delimiter(delimiter) = &flat[10] {
    assert_eq!(delimiter.kind(), DelimiterKind::Bracket);
    assert_eq!(delimiter.position(), DelimiterPosition::Close);
} else {
    panic!("expected delimiter");
}

Features

• proc-macro: Enables feature proc-macro of proc-macro2 (enabled by default).
• span-locations: Enables feature span-locations of proc-macro2 (enabled by default).

Structs

Delimiter	Single delimiter token.
FlattenRec	Iterator adaptor that recursively flattens the token trees of an IntoIter.

Enums

DelimiterKind	The kind of a delimiter.
DelimiterPosition	Whether a delimiter is opening or closing.
Token	Single atomic token.

Traits

IntoFlattenRec

Allows conversion into FlattenRec.