docextract
Extract //// documentation from Rust source files.
docextract is a tiny Rust library and CLI for pulling dedicated //// comments out of Rust
source files into a standalone Markdown document. It is the missing piece for documenting
declarative macro DSLs and generated APIs — things rustdoc cannot see — and it does this
with a strict line-scanning approach: no Rust parsing, no token trees, no macro expansion,
and zero dependencies by default (an optional regex feature adds regex-lite for advanced
pattern matching). A build.rs step (or the CLI) extracts these comments into a standalone
Markdown file that ships with your crate or lives alongside your documentation.
Note: This crate was developed with AI assistance.
Documentation Syntax
What counts as a documentation line
A line is a documentation line when it starts with optional ASCII whitespace (spaces or tabs),
followed by exactly four slashes ////, followed by a space (0x20), a tab (0x09), or the end
of the line.
When //// appears at the end of a source line (preceded by a space or tab), the line becomes
a code-capturing line. The code on the left is captured, and {{...}} placeholders in the
text on the right are substituted with the captured code.
Valid documentation lines:
//// Hello space separator
//// Hello extra spaces are content
//// Hello tab separator (tab after the slashes)
//// empty doc line → verbatim blank line
//// Hello indented with spaces
//// Hello indented with a tab
let x = 42; //// {{}} code-capturing line — {{}} substitutes the code
Tag directives
When a //// line has no separator (the character immediately after the slashes is not a
space, tab, or newline), it is a tag directive:
////chapter1 persistent tag switch — all following doc lines carry this tag
////glossary definition transient tag — this line only, then tag reverts
The initial tag is the filename without its extension (e.g., lib for src/lib.rs). Tags
enable filtering and reordering — see Tagged
Documentation for details.
Extraction rules
For each documentation line, the extractor removes, in order:
- Leading ASCII whitespace — spaces and tabs before the slashes.
- The four slashes —
////. - Exactly one separator character — the space or tab immediately after the slashes.
Everything after the separator is preserved verbatim — no Markdown interpretation, no whitespace normalization.
The one-separator rule is what makes indentation inside your documentation possible. //// Indented code block keeps the second space, so code blocks and nested lists survive
extraction intact.
{{...}} in regular doc lines is literal. Placeholder substitution only happens on code-capturing lines (where //// is at the end). A regular //// {{}} doc line emits {{}} verbatim. To use placeholders, write code before the ////:
//// {{}} → emitted as "{{}}" (verbatim — regular doc line)
let x = 42; //// {{}} → emitted as "let x = 42;" (code-capturing line — placeholder resolved)
Blocks
Consecutive documentation lines form a documentation block. No blank line is automatically
inserted between blocks. To separate blocks visually, write an explicit //// line between
them. Non-documentation source lines never appear in the output, except for code-capturing
lines (lines ending with ////) which emit the resolved template text.
Source:
//// First
//// Second
let x = 5;
//// Third
Output:
First
Second
Third
Blank lines inside a block
To keep a blank line inside a block, write an empty //// line (or //// — both are
equivalent):
//// First
////
//// Second
Output:
First
Second
Line endings
Line endings are preserved from the source. A file using CRLF produces CRLF output; a file using LF produces LF output.
Multi-file output
No blank line is added between files — output from multiple inputs is plain concatenation of each file's blocks, in the order given.
Markdown is your job
docextract does not interpret Markdown. It copies text verbatim. Write //// # Heading
(with the space) and # Heading lands in the output. Full Markdown rendering is a non-goal —
the tool is strictly an extractor.
Code-Lifting Syntax
Non-documentation source lines ending with //// capture the code on the left side for
placeholder substitution in the doc text on the right side.
Placeholders:
{{}}— substitutes the captured code with leading and trailing whitespace stripped{{N}}— removes the first N UTF-8 characters, substitutes the remainder{{/regex/}}— finds regex in the code, substitutes the match (extract mode){{/regex/subst/}}— matches regex and outputs only the expanded substitution (using$1,$2backreferences). Non-matched text is discarded. No match → empty output.
Any ASCII punctuation can be the delimiter: {{!regex!subst!}}
Examples:
let x = 42; //// {{}}
// let x = 42;
let x = 42; //// The answer is {{}}.
// The answer is let x = 42;
let foo=bar; //// {{8}}
// foo=bar;
let x = 42; //// {{/let ([[:alnum:]]+) = (.+);/$1 is $2/}}
// x is 42
Code-capturing lines produce regular persistent documentation entries. They carry the current
tag and respect all filtering rules. The {{}} and {{N}} forms work without any feature
flags; the sed-like forms ({{XregexX}} / {{XregexXsubstX}}) require the regex feature
(enabled by default).
Capture zone continuation
Lines ending with //// establish a capture zone at the indentation of the captured
code. Within this zone, //// lines at deeper indentation are treated as code-capturing
suffixes (not doc prefixes) — the leading whitespace becomes empty captured code, and the
template resolves normally:
code; //// {{}} ← establishes capture zone at indent 4
//// {{4}} suffix ← deeper indent — continuation, {{4}} on empty → "", appends " suffix"
more; //// {{}} ← same indent — regular code-capturing line, zone stays open
//// end ← indent 0 ≤ 4 — zone resets, regular doc line
Produces:
code;
suffix
more;
end
The zone ends when a line at ≤ the zone's indentation is encountered (non-code-capturing
lines, tag directives, or //// lines at the zone indent or shallower).
Build Script Integration
Build scripts are the primary use case: extract macro DSL documentation into OUT_DIR on
every build, then embed the generated Markdown in your crate's rustdoc output.
Setup
# Cargo.toml
[]
= "0.1"
// build.rs
use env;
Feature flags
The regex feature is enabled by default and adds regex-lite as a dependency, providing
full regex pattern matching for tag filters. This is the recommended configuration for most
users.
To opt into zero dependencies (no regex-lite), disable default features:
# Cargo.toml
[]
= { = "0.1", = false }
With default-features = false, pattern matching uses the lightweight hand-rolled matcher
(see Pattern syntax). Tag filtering and reordering work identically — only
the pattern syntax changes.
Embedding the output in your docs
Include the generated Markdown in your crate's rustdoc output via a hidden module whose doc comment is the extracted file:
Because rustdoc never sees //// comments, they cannot leak into the crate's public API docs;
the extracted page is attached to the hidden module instead.
Automatic cargo:rerun-if-changed
Docextract::new() checks for OUT_DIR. When it is set (build script context), each
.input() call automatically emits cargo:rerun-if-changed={path} on stdout — no manual
println! needed. If your macro source changes, Cargo reruns the build script and regenerates
the docs. This eliminates the classic stale-build bug of forgetting the directive.
For multi-file builds, use a loop — each .input() call emits its own directive:
use env;
Overriding auto-detection
Docextract::new() auto-detects build script context: if OUT_DIR is set, rerun_if_changed
defaults to true and directives are emitted. Outside a build script (OUT_DIR unset),
directives are off by default.
Use .rerun_if_changed() to override:
#
#
Tagged Documentation (Filtering & Reordering)
docextract supports tagging documentation lines to enable filtering and reordering in the
output. Tag directives use ////tagname syntax — a line with exactly four slashes followed by
a name (no space).
Default tags
Every documentation line has a tag. The initial tag is the filename without its extension. In
src/lib.rs, untagged lines get the tag lib:
// In src/lib.rs — initial tag is "lib"
//// This line has tag "lib"
////chapter1
//// This line has tag "chapter1"
Tagging lines
A bare ////tagname is a persistent tag switch — all subsequent //// lines carry this
tag until another switch. The directive line itself produces no output.
A ////tagname some text is a transient tag — the content some text is emitted under
tag tagname, and the previous tag resumes. Transient lines are useful for injecting
glossaries, lists, or footnotes.
////chapter1
//// # Getting Started
////glossary route — a URL path bound to a handler
//// This is still under chapter1
Blocks and tags
Consecutive documentation lines under the same tag form a block. Blocks are concatenated
without auto-separators — use an explicit //// line to insert a blank line.
Source:
////chapter1
//// # Getting Started
////
//// This blank line was explicit.
////glossary
//// A definition.
//// More text under glossary.
Output:
# Getting Started
This blank line was explicit.
A definition.
More text under glossary.
A persistent tag switch (////newname) ends the current block and starts a new one under the
new tag. Non-documentation lines also end blocks. No blank line is ever emitted between blocks
automatically — the author controls all spacing.
Filtering input
Use .input_filtered(path, pattern) to extract only lines whose tag matches a pattern:
#
Filtering output
Use .filter_output(pattern) to apply a global filter across ALL inputs:
#
Input filters and the output filter combine with AND logic — a line must pass both.
Reordering
Read the same file multiple times with different filters to reorder sections:
#
The file is read and parsed once; subsequent filtered reads operate on the cached content.
Pattern syntax
By default, patterns use full regex-lite syntax — POSIX character classes, grouping,
quantifiers, and more:
[[:alpha:]]([[:alnum:]])*_bar // matches a1_bar, a123foo_bar
chapter.* // matches chapter1, chapter_intro
When built without the regex feature (--no-default-features), a lightweight hand-rolled
matcher is used instead, supporting this subset:
| Pattern | Meaning |
|---|---|
.* |
Matches zero or more characters |
.? |
Matches zero or one character |
[abc] |
Character class — matches a, b, or c |
[^abc] |
Negated character class |
api |
Literal match |
The hand-rolled matcher supports no POSIX classes, no () grouping, and no +/{n,m}
quantifiers. Patterns written for the reduced syntax work identically on both paths.
Transient tags in unfiltered output
Transient tags are excluded from unfiltered output — they only appear when an active filter
matches their tag. This preserves backward compatibility: the transient tag directive
////glossary term definition is treated like the old silent-ignore behavior when no filters
are active.
Example project
The demo-macro example demonstrates the full docextract pipeline:
- A Rust crate uses
////comment syntax to document a declarative macro DSL build.rsextracts those////comments into$OUT_DIR/macro-docs.md- The extracted docs are re-integrated into
cargo docoutput via#[doc = include_str!(...)] - The
define_routes!macro and its generated dispatch function are exercised by unit tests
How It Works
Source documentation (src/lib.rs): The crate uses //// comment syntax throughout to
document the define_routes! macro syntax, generated code patterns, usage examples, and
tests. These comments are invisible to rustdoc — keeping macro DSL documentation separate from
the public API surface.
Build script (build.rs):
#
On every build, build.rs scans src/lib.rs for //// doc lines and writes the extracted
content to $OUT_DIR/macro-docs.md. The builder automatically emits cargo:rerun-if-changed
directives.
Rustdoc integration (src/lib.rs):
This embeds the extracted documentation into a dedicated module page in cargo doc output.
Building and Testing
CLI Usage
Invocation
cargo-docextract <output> [inputs...]
<output>— mandatory, must be the first argument. A file path, or-for stdout.[inputs...]. Zero or more file paths. A file path, or-for stdin.
There are no options and no flags, the interface is positional only. -o, -V, --,
and similar are treated as file paths. The single exception is --help, which triggers the
help text from any position.
Stdin and stdout
-as the output writes to stdout.-as an input reads from stdin.- A lone
-(cargo docextract -) is shorthand forcargo docextract - -: it reads stdin and writes stdout — a pipe pass-through filter.
Help
Invoking with zero arguments or with --help anywhere prints the help text and exits
0:
cargo-docextract - extract //// documentation from Rust source files
Usage: cargo-docextract <output> [inputs...]
Arguments:
<output> Output file path, or '-' for stdout (mandatory).
Prefix with 'filter@' for output filtering,
e.g., 'chapter.*@output.md'.
[inputs...] Input file paths, or '-' for stdin (zero or more).
Prefix with 'filter@' to filter by tag pattern,
e.g., 'api@src/lib.rs'.
Examples:
cargo-docextract docs.md src/macros.rs Extract from one file
cargo-docextract - src/macros.rs Extract to stdout
cargo-docextract api@docs.md src/lib.rs Filter by 'api' tag
cargo-docextract 'chapter.*'@out.md chapter1@in.rs chapter2@in.rs
cargo-docextract - Pipe stdin → stdout
cargo-docextract --help Print this help
There are no options or flags. All arguments are positional.
Exit codes
| Code | Meaning |
|---|---|
| 0 | Success — all inputs processed, output written. Also used for help and for an empty input list. |
| 2 | I/O error — output file cannot be created, an input cannot be read, a flush fails, or no input file could be read at all. |
There is no exit code 1: with zero arguments the CLI prints help (exit 0), and with any arguments the first argument is the output — a "missing output" condition cannot arise.
Common invocations
# Unfiltered extraction (backward compatible)
cargo-docextract docs.md src/lib.rs src/macros.rs
# Output filtering — emit only lines tagged with a chapter.* pattern
cargo-docextract 'chapter.*'@chapters.md src/lib.rs
# Input filtering — only extract api-tagged lines from one file
cargo-docextract docs.md api@src/macros.rs src/lib.rs
# Reordering — read same file with different filters
cargo-docextract out.md glossary@src/macros.rs chapter1@src/macros.rs api@src/macros.rs
# Pipe stdin through, filter output to stdout
cargo-docextract 'api.*'@- -
Messages
All messages go to stderr, prefixed with cargo-docextract::
cargo-docextract: warning: input file not found: missing.rs
cargo-docextract: error: output.md: Permission denied (os error 13)
cargo-docextract: error: src/macros.rs: Permission denied (os error 13)
cargo-docextract: error: stdin: broken pipe (os error 32)
cargo-docextract: error: output: No space left on device (os error 28)
cargo-docextract: error: no input files could be read
Behavioral notes:
- File not found is a warning, not an error. Processing continues with the remaining inputs. The exit code reflects the overall outcome: 0 if at least one input was processed, 2 only if no input could be read.
- Other I/O errors abort immediately with exit code 2.
-reads stdin exactly once. If-appears twice, the second occurrence is a no-op.- An empty input list is well-defined: the output file is created empty, exit 0.
License
Licensed under the WTFPL (Do What The Fuck You Want To Public License), Version 2, December 2004.
Repository: https://seed.pipapo.org/nodes/seed.pipapo.org/rad:z446u5iBrxqbKrUrGWAuNSWkJD3PL