Module roxygen 

Roxygen sub-lexing and classification helpers. Low-level surface consumed by the arity crate; not covered by semver stability guarantees. Roxygen2 doc-comment recognition, sub-tokenization, and block structure.

A roxygen line is a comment whose text matches ^#+' (one-or-more # followed by a single '). Such lines are sub-tokenized—rather than emitted as one COMMENT token—so their structure (marker, tags, arguments, prose) lives directly in the lossless CST. The sub-tokens’ texts tile the line’s bytes exactly, preserving the round-trip invariant.

The work is split into three phases, one module each, plus this parent which owns the macro-classification layer (the \macro arity/verbatim tables that both the lexer and the structure builder consult) and the shared balanced-delimiter scan:

• [lex] — sub-lexing: block-mode resolution + the per-line tokenizer (text → Vec<Token>).
• [group] — block grouping: wrapping a run of lines in a ROXYGEN_BLOCK and laying out its section/paragraph skeleton (Vec<Token> → Vec<Event>).
• [build] — structure building: the block-level Rd-macro and markdown constructs (\itemize{…}, \describe{…}, markdown lists) dispatched from the grouper.

Enums

MdArgPiece

One piece of a structural Rd-macro argument fed to resolve_md_inline_pieces: either raw prose Text (markdown-lexed) or a pre-parsed nested Macro (its raw \name{…}/\name source, kept opaque).

Functions

advance_md_col

Advance a markdown value column across one whitespace character: a tab expands to the next 4-column tab stop (CommonMark’s rule — roxygen2 hands cmark the tag value with tabs intact), any other character is one column. Columns are zero-based value coordinates: column 0 is the first character after the #' sigil plus the one whitespace character roxygen2’s tokenizer strips (consumeWhitespace(1), parser2.cpp — a space or a tab). The single source of column arithmetic for the block builder’s indent gauge and the projector’s column stripping.

is_argument_taking_rd_macro

Whether parse_Rd consumes {…} argument groups for \name — a built-in (is_known_rd_macro) or a system user macro ([SYSTEM_RD_USER_MACROS]). Everything else is an unrecognized \word: parse_Rd tags it UNKNOWN and consumes nothing, so a written group is a following sibling brace list (\zzz{x} → (UNKNOWN "\zzz") (LIST (TEXT "x"))), exactly like a zero-arity call’s. Arity still applies on top — a zero-arity name in either set takes no group either.

is_block_rd_macro

Whether the macro named name (without the leading \) is block-level — a display construct that owns its own lines rather than inline markup inside a prose run. See [BLOCK_RD_MACROS]. pub for the formatter, which keys a macro’s layout on it (semver-loose, see the crate docs).

is_fragile_for_md

Whether the macro named name (without the leading \) has its {…} content protected from markdown under @md (roxygen2’s escaped_for_md). A fragile macro keeps its argument literal; a non-fragile one has it markdown-processed. See [FRAGILE_FOR_MD_RD_MACROS].

is_known_rd_macro

Whether name (without the leading \) is a built-in Rd macro parse_Rd recognizes. The single source of truth for the known/unknown split, shared by the lexer (gate brace-less recognition) and the projector (name-only → (\name) vs (UNKNOWN …)). See [KNOWN_RD_MACROS].

is_line_significant_rd_macro

Whether the macro named name (without the leading \) renders its body’s line breaks, so a soft wrap inside it must be preserved rather than joined. See [LINE_SIGNIFICANT_RD_MACROS]. pub for the formatter, which keys reflow on it (semver-loose, see the crate docs).

is_md_standalone_html_tag

Whether text is a complete standalone HTML tag — one open/closing/ self-closing tag followed by nothing but spaces or tabs. Placed alone at a markdown line’s content start at a fresh (non-paragraph) position, such a line opens an HTML block (CommonMark start condition 7, blank-terminated). Condition 7 cannot interrupt a paragraph, so mid-paragraph line starts are safe — the formatter guards only a paragraph’s first line with this.

is_multi_arg_rd_macro

Whether the macro named name (without the leading \) takes more than one {…} argument group — parse_Rd’s structural macros, whose arguments are list wrappers. See rd_macro_arity.

is_rd_braceless_drop_macro

Whether a brace-less \name in prose is dropped by parse_Rd’s recovery with the surrounding text continuing as plain TEXT (the “unexpected TEXT/section header, expecting {” recovery): an argument-taking, non-zero-arg macro outside [is_sticky_braceless_rd_macro]. Zero-arg names are complete calls (carved by the lexer), unknown names are (UNKNOWN …) nodes (also carved), and the sticky set leaves parse_Rd in a code/verbatim mode. Of that set, brace-less \item is projected as an (UNKNOWN "\item") node (split_braceless_items in the projector) and the code/verbatim mode-flip names swallow the section (split_sticky_braceless_swallow). Mode-independent: the @md pipeline is a net no-op on a backslash run before a letter, so parse_Rd sees the same brace-less macro either way.

is_table_delim_row

Whether line is a GFM table delimiter row: it contains at least one unescaped | (so a bare ---, which is a setext underline, is not a single-column table) and every cell (trimmed) is :?-+:? (optional leading colon, one or more hyphens, optional trailing colon). The pipe requirement mirrors cmark-gfm, which treats a pipeless dash run as a setext heading.

is_verbatim_rd_macro

Whether the macro named name (without the leading \) takes verbatim {…} content. Used both when building the CST (don’t recurse into a verbatim body) and when projecting it (emit VERB, not coalesced TEXT) — the projector’s block-form verbatim arm keys on this set, so it is pub like its rd_macro_arity sibling (semver-loose, see the crate docs).

md_fence_run_closes

Whether the fence line closer closes a code block opened by the fence line opener (both given as their fence-leaf text, starting at the fence run; indentation is checked by the caller in its own coordinate system): the same fence character, a run at least as long as the opener’s, and nothing but whitespace after the run — a closing fence has no info string (CommonMark 4.5). A non-matching fence line is content.

md_ws_gauge

The one-based indent gauge of a marker→content whitespace run: the first character (space or tab alike) is the one roxygen2 strips and maps to gauge 1 (value column 0); the rest expand via advance_md_col, so the gauge is the content’s value column plus one. An empty run gauges 0. For an all-space run this equals the character count — the pre-tab behavior.

rd_macro_arity

How many {…} argument groups the macro named name (without the leading \) takes; 1 for every name outside [RD_MACRO_ARITY]. Drives the lexer (consume that many groups into one token), the tree builder (emit them as children), and the projector (each group of a multi-argument macro is a list argument — a multi-atom one becomes a (GRP …)).

rd_macro_name_end

The end index of an Rd macro name starting at bytes[start] (the byte after the leading \). An Rd command name is [A-Za-z][A-Za-z0-9]*: a leading letter then any letters or digits (e.g. \linkS4class). Returns start when no valid name begins there (\\, \{, \4, end of input). The single source of truth for where a \name ends, shared by the lexer and the tree builder.

resolve_md_inline

Resolve a bare prose content string as a @md markdown inline run and return the resulting ROXYGEN_PARAGRAPH node, whose children are the resolved inline elements (text, emphasis/strong nodes, links, code spans, nested Rd macros). Drives the projector’s translation of a non-fragile Rd macro’s argument under @md (\emph{*x*} → \emph{\emph{x}}): the projector slices out the raw argument text and feeds it here, reusing the real inline pass (the delimiter-stack arena) rather than a second markdown scanner — so nesting, links, and code spans resolve exactly as in ordinary @md prose. A nested fragile macro stays an opaque ROXYGEN_RD_MACRO token here; the projector keeps its argument literal by recursing with the same fragility check.

resolve_md_inline_pieces

Resolve a structural Rd-macro argument (\item/\tabular/\href under @md) as a single markdown inline run from its already-carved pieces, returning the resolved ROXYGEN_PARAGRAPH node (same shape as resolve_md_inline).

resolve_rd_inline

Non-markdown twin of resolve_md_inline: lex content as a plain Rd prose fragment (no markdown recognizers; \name{…} macros still carve, gated by the backslash-pairing parity) and return the ROXYGEN_PARAGRAPH node. Drives the projector’s re-parse of a demoted markdown span’s rendered body, which tools::parse_Rd reads as ordinary Rd text once the generating macro’s backslash has been paired away.

split_table_row_cells

Split a GFM table row into its cells, honoring backslash-escaped pipes. One optional leading and one optional trailing unescaped | are stripped (the GFM leading/trailing pipe), then the remainder is split on each unescaped |. Cells are returned untrimmed (callers trim). An escaped \| stays inside its cell. Shared by the recognition gate (cell counting) and the projector (cell rendering) so the two never disagree on where a cell begins.

starts_md_html_block

Whether text placed at a markdown line’s content start would open an HTML block (any of the start conditions 1–6 [scan_md_html_block] recognizes). HTML blocks interrupt a paragraph, so the formatter must never reflow such text onto a line start — an inline comment/PI/CDATA/declaration/block-tag (or literal prose that merely looks like an opener, e.g. an unterminated <!--) would reparse as a block and change the rendered Rd.

sticky_braceless_code_mode

For a brace-less sticky Rd macro name (\code/\verb/\doi/…, see [is_sticky_braceless_rd_macro]), which parse_Rd lexer state its “expecting {” recovery leaves in place: Some(true) for the R-code macros (the recovery stays in R-code mode → per-line RCODE atoms) and Some(false) for the verbatim macros (verbatim mode → per-line VERB atoms). None for a non-sticky name and for item, whose out-of-list misuse is instead an (UNKNOWN "\item") node (handled by the projector’s split_braceless_items). The two built-in sets are exactly [STICKY_BRACELESS_RD_MACROS] minus item, probed against R 4.5 (see that list’s doc); the argument-taking system user macros are uniformly verbatim, probed against R 4.6.

system_rd_user_macro_names

Every name in [SYSTEM_RD_USER_MACROS], for the projector’s drift check against its own definition table.