Module token 

token — the public seam on both sides

What a producer hands the compiler, and what a renderer hands back.

The private capture/ and generation/ homes own the two directions while this module keeps the established public token paths unchanged. The shared rendered-path magnitude lives at this seam, while each declaring home retains its established public compatibility constant.

Why it is typed

proc_macro is a proc-macro-crate-only API. A crate that is not compiled as a proc macro cannot name its types at all, so the compiler — which is ordinary callable Rust and must stay so — can neither take a TokenStream nor hand one back.

The answer is not to fall back to strings. A string is a token stream with its structure thrown away, and everything the capture then has to do is re-derive structure the compiler already had.

So the seam is typed on both sides.

Reading

CapturedTokenTree is one token of a declared input: a payload, a stable TokenPath naming exactly where it sits in the tree, and an opaque SpanHandle indexing the producer’s own table. Delimited groups stay groups, including a compiler’s invisible group. Compiler-host input is never re-lexed.

CaptureBuilder is the only mint of a complete captured input. A producer supplies its own source position and a CapturedAtom, or opens a nested group; the builder issues the path and handle, spends every declared magnitude, retains positions in handle order, and derives the final denominator from that roster. No producer can state the resulting capture facts as sibling arguments. A capture operation consumes its open CaptureLevel, and only a successful operation returns the level, so a refused partial tree has no road to CaptureLevel::finish. The refused attempt’s positions remain available for its diagnostic; opening a fresh level rolls back that attempt while preserving handles issued by earlier successful captures in the same table. Where a producer refuses one token after issuance, the refusal retains both the declaration-local path and the producer-local handle.

A payload carries a literal’s value and never its spelling. "x" and r"x" are one text, "a\nb" is three characters, and which prefix a producer read is not a fact the tree keeps. capture_literal is where a lexed spelling becomes that value, and it refuses a form it has no row for rather than filing it under a neighbouring one.

Token distinctions that change a proc macro’s input remain distinct: ordinary and raw identifiers have different rows, punctuation retains whether it joins what follows, lifetimes remain quote-plus-identifier tokens, and invisible compiler groups remain groups. Whitespace and ordinary comments do not enter the normalized declaration. Doc comments enter as the doc attributes the compiler presents to a proc macro.

Every producer walks under the same captured-input magnitudes — depth, level, whole tree, and capture work — declared as plain constants beside the capacities they govern. A producer that skips or backtracks charges that observation through CaptureLevel::examined, so work discarded before capture does not disappear from the budget. A SpanHandle means “the token at this index of the table the producer built while capturing”. The compiler never resolves one; it carries the handle into a diagnostic so that whoever produced the input can map it back to the exact compiler span, which is what puts a compile_error! on the offending token rather than on the first token of the declaration.

TextCapture::read is the third producer. A compiler is one, a test is another, and text is the third — it exists so that the reproduction route a diagnostic names is a real road and not a promise. It uses a pinned low-level compiler lexer for token boundaries and slices every spelling from the original source by the lexer-reported byte range. Its source-byte magnitude is checked before lexing and stands independently of the capture tree and work magnitudes. The CapturedInput it returns is the shared normalization boundary against which the compiler-token producer is observed.

CaptureCursor reads generic mechanical shapes from that boundary without interpreting them. It leaves declaration vocabulary and diagnostic policy with the caller, and its typed refusals retain the exact token span available where a word, identifier, punctuation seat, group, separator, magnitude, or end condition disagrees. This is the reusable parser battery for subject-owned grammars, not a grammar owned by Macroonz.

CapturedFragment is one borrowed exact run inside that same boundary. A cursor can return the fragment consumed by a caller-owned structural read, so types, paths, signatures, expressions, patterns, generic clauses, bodies, and other exact Rust seats retain their captured tokens and source handles without becoming strings.

AuthoredItem is the shallow structural lens over one supported complete caller-authored item boundary. It identifies the outer attributes, visibility, qualifiers, structural item family, optional name, generics, where clause, signature, and body group while the complete CapturedFragment remains the one preserved token reading. The lens is not a Rust AST, and Rustc remains responsible for full syntax, type, lifetime, ownership, and coherence judgments.

Writing

GeneratedTree is what a renderer produces. A renderer states a literal’s value and never its spelling here too: the quoting, the escaping, and the absence of a suffix belong to the tree. That is what keeps b"…" from being assembled out of a word and a quoted string — two tokens where the address reading it matches one — and what lets one count be written into a u32 seat, a u64 seat, and a usize seat, because an unsuffixed literal is typed by the position it lands in.

The private generation home’s composers are the rest of what a renderer needs: paths, calls, method chains, bindings, constants, functions, attributes, rosters. A renderer states what it means and never assembles punctuation by hand. The keyed slice projectors walk an informed KeyedRoster or KeyedRosterAssignment in its structural order while the renderer supplies every row’s tokens and the ordinary Rust item surrounding the slice.

Where generated output must repeat exact caller-authored Rust, CapturedFragment::generated projects the captured token structure directly into the generated vocabulary. The road never builds and reparses source text, retains invisible compiler groups, carries each producer span as nonsemantic emission provenance, and uses a guarded GeneratedLiteral for numeric, character, byte, and C-string forms that require exact literal custody. Producer spans do not enter equality, hashing, canonical bytes, or human inspection, and only the compiler host resolves them while materializing the declaration-site token stream.

The written roster grows only at its end. Each arm’s stable slot is one byte of the tree’s canonical encoding, and those bytes are what a rendered unit’s identity is derived over. An ordinary identifier and a raw identifier occupy distinct rows, while every pre-existing row keeps its occupied slot.

What it is not

The Rust source text a person reads is GeneratedTree::inspected — a projection of the tree, produced for a person, never the artifact. Nothing parses it back and no identity is derived from it.

Nothing here knows what a declaration means. The seam carries tokens with their structure and their spans intact; the grammar written in them is the caller’s.

Structs

AuthoredItem

A checked structural lens into one supported complete caller-authored Rust item.

AuthoredItemReadRefusal

One refused authored-item lens with the exact available producer span.

CaptureBuilder

The only state that issues capture handles and retains the producer’s matching source positions.

CaptureCursor

A read cursor over one normalized captured-token sequence.

CaptureLevel

One nesting level borrowed from a CaptureBuilder.

CaptureReadRefusal

One refused mechanical read with the exact producer span available at that site.

CaptureWalk

The running state of one capture walk: what the walk has spent, and how much of the whole-tree magnitude it has taken.

CapturedFragment

One borrowed run of captured tokens under its original source boundary.

CapturedInput

One captured declared input: the top-level token trees, and how many span handles the producer issued.

CapturedTokenTree

One captured token: what it carries, where it sits, and how to reach the compiler span it came from.

FragmentGenerationRefusal

One refused captured-fragment projection with the exact source span it belongs to.

GeneratedLiteral

One exact literal form carried from declared Rust into generated Rust.

GeneratedRowRefusal

Why one flat keyed-row projection could not produce exactly one non-empty item run.

GeneratedTree

One generated token tree: the artifact a renderer produces.

SourceCoordinate

One compiler-local source position with its coordinate system stated.

SpanHandle

An opaque index into the producer’s span table.

SpanResolutionRefusal

Why one span table could not say where a handle sits.

TextCapture

One declared input read from text: the captured trees, and the byte offsets that resolve every handle the read issued.

TextReadRefusal

One refused text read: the established cause, and the byte it sits at.

TokenPath

Where one captured token sits, as the index route from the root of the declared input.

Enums

AuthoredItemKind

The structural Rust item family one authored-item lens recognized.

AuthoredItemReadIssue

Why one captured input could not provide an authored-item structural lens.

CaptureBound

Which declared magnitude one capture ran past.

CaptureBuildRefusal

Why a checked capture was not completed.

CaptureExpectation

The mechanical token shape one capture read asked for.

CaptureReadIssue

Why generic captured-token grammar mechanics could not complete one read.

CapturedAtom

One non-group value a capture producer offers to the checked builder.

CapturedDelimiter

The delimiter one captured group is written with.

CapturedPayload

What one captured token carries.

CapturedSpacing

Whether one punctuation seat is joined to what follows or stands alone.

CoordinateRole

The coordinate system one source position is counted in.

FragmentGenerationIssue

Why one preserved captured fragment could not become a generated tree.

GeneratedDelimiter

The delimiter one generated group is written with.

GeneratedLiteralRefusal

Why an exact literal could not be admitted for generation.

GeneratedSpacing

Whether one generated punctuation mark joins the token after it.

GeneratedToken

One token a renderer writes.

LiteralReadCause

Why one literal spelling could not be read into the value it names.

SpanTable

How a producer answers “where is the token this handle names?”.

TextLexicalCause

Why the low-level lexer could not normalize one spelling.

TextReadCause

Why one text read refused.

Constants

CAPTURED_TOKEN_LIMIT

Token trees one captured input may carry at any one nesting level.

CAPTURED_TREE_TOKEN_LIMIT

Tokens one captured input may retain across the whole tree, and positions one span table may hold.

CAPTURE_WORK_LIMIT

Units of capture work one walk may spend, one unit per examined token.

GENERATED_TOKEN_LIMIT

Tokens one generated tree may carry at any one nesting level.

TEXT_SOURCE_BYTE_LIMIT

Source bytes one text capture may read before tokenization.

TOKEN_PATH_DEPTH_LIMIT

Steps one token path may carry, and so how deeply a declared input may nest.

Functions

absolute_path

The absolute path ::a::b::c.

and_all

Every comparison joined by &&.

associated_constant

One associated constant declaration or definition.

associated_function

One associated function declaration or definition from an exact signature.

associated_type

One associated type declaration or definition.

attribute

One attribute #[body].

bound_local

One statement let name = expression;.

bound_path

The path root::a::b::c, rooted at a crate the caller named.

call

One call path(arguments).

capture_literal

Read one literal token’s spelling into the atom it names.

comma

Two token runs separated by a comma.

comma_many

Every token run separated by a comma, with no trailing one.

constant

One item const name: kind = value;.

consuming_receiver

The conventional consuming receiver self.

decorated

Prefix one item with its declared attributes and visibility.

documentation

One documentation attribute over one sentence.

enumeration

One enum enum Name<...> where ... { variants }.

equality

One comparison left == right.

exclusive_receiver

The conventional exclusive receiver &'a mut self, with an empty lifetime run producing &mut self.

function

One conventional fn name(parameters) -> result { body } item.

function_item

One function or method item from an exact signature and exact body.

function_signature

One exact function signature without visibility, attributes, body or terminal semicolon.

generic_parameters

One generic-parameter list <parameters>, or nothing when the roster is empty.

group

One delimited group.

implementation

One inherent or trait implementation with exact qualifiers, parameters, target, predicates and items.

inline_module

One inline module mod name { items }.

keyed_assignment_items

Projects one exact keyed assignment into a flat item run in denominator order.

keyed_assignment_slice

Projects one exact keyed assignment into a borrowed Rust slice expression in denominator order.

keyed_roster_items

Projects one keyed roster into a flat item run in retained order.

keyed_roster_slice

Projects one keyed roster into a borrowed Rust slice expression in retained order.

match_arm

One match arm with an optional exact guard and a trailing comma.

match_expression

One match expression over exact subject and arm runs.

metavariable

One macro metavariable, as the two tokens that spell it.

method_call

One call receiver.method(arguments).

method_chain

One chain receiver.first().second().third(), every method taking no argument.

named_field

One named field name: kind.

named_struct

One named struct struct Name<...> where ... { fields }.

named_variant

One named variant.

pinned_receiver

The conventional pinned receiver self: ::core::pin::Pin<&'a mut Self>.

rendered_identifier

Whether one spelling is a single Rust identifier a rendering is willing to write.

rendered_name

Whether one spelling can NAME a rendered item: a single identifier the language has not already taken.

result_type

The type ::core::result::Result<ok, error>.

roster

The ::std::vec![…] a roster-taking constructor is handed.

rust_keyword

Whether one spelling is a Rust keyword no rendered item can be named by.

shared_receiver

The conventional shared receiver &'a self, with an empty lifetime run producing &self.

text_pair

Two spellings as the two comma-separated text literals a two-argument parser takes.

trait_declaration

One trait declaration with exact qualifiers, parameters, supertraits, predicates and associated items.

tuple_struct

One tuple struct struct Name<...>(fields) where ...;.

tuple_variant

One tuple variant.

twin_path

The path $binding::a::b::c, rooted at a metavariable a macro shell will bind.

type_alias

One type alias type Name<...> = value where ...;.

typed_parameter

One typed parameter pattern: kind.

unit_struct

One unit struct struct Name<...> where ...;.

unit_variant

One unit variant.

use_item

One import use path [as alias];.

where_clause

One where clause over exact caller-owned predicates, or nothing when the roster is empty.