Function token_stream_to_consistent_string 

pub fn token_stream_to_consistent_string(tokens: TokenStream) -> String

Available on crate feature _helpers only.

Helper utilities for building procedural macros Converts a token stream to a consistent string representation without spaces.

The standard TokenStream::to_string() method can produce inconsistent output depending on the context - sometimes including spaces between tokens, sometimes not. This function ensures a consistent, space-free representation that’s reliable for comparisons, hashing, or other operations requiring deterministic output.

Context-Dependent Behavior of TokenStream::to_string()

• When operating on syn::File: spaces generally appear between tokens
• Inside procedural macros: spaces generally don’t appear
• Other contexts: behavior may vary

Arguments

• tokens - The token stream to convert to a consistent string

Returns

A string representation with no spaces between tokens, ensuring consistent output regardless of the original context

Examples

let tokens = quote! { fn hello() -> String { "hello world".to_string() } };
let result = token_stream_to_consistent_string(tokens);
assert_eq!(result, "fnhello()->String{\"hello world\".to_string()}");

Token Processing

The function handles different token types:

• Groups: Processes delimiters ((), {}, []) and recursively processes contents
• Identifiers: Removes leading/trailing whitespace
• Punctuation: Removes leading/trailing whitespace
• Literals: Removes leading/trailing whitespace

Use Cases

• Comparing token streams for equality regardless of spacing
• Debugging token streams with predictable output