docextract
Extract //// documentation from Rust source files.
docextract is a tiny Rust library and CLI for pulling dedicated //// comments out of Rust source files into a standalone Markdown document. It is the missing piece for documenting declarative macro DSLs and generated APIs — things rustdoc cannot see — and it does this with a strict line-scanning approach: no Rust parsing, no token trees, no macro expansion, and zero dependencies by default (an optional regex feature adds regex-lite for advanced pattern matching). A build.rs step (or the CLI) extracts these comments into a standalone Markdown file that ships with your crate or lives alongside your documentation.
Note: This crate was developed with AI assistance.
Documentation Syntax
What counts as a documentation line
A line is a documentation line when it starts with optional ASCII whitespace (spaces or tabs), followed by exactly four slashes ////, followed by a space (0x20), a tab (0x09), or the end of the line.
Valid documentation lines:
//// Hello space separator
//// Hello extra spaces are content
//// Hello tab separator (tab after the slashes)
//// empty doc line → verbatim blank line
//// Hello indented with spaces
//// Hello indented with a tab
Tag directives
When a //// line has no separator (the character immediately after the slashes is not a space, tab, or newline), it is a tag directive:
////chapter1 persistent tag switch — all following doc lines carry this tag
////glossary definition transient tag — this line only, then tag reverts
The initial tag is the filename without its extension (e.g., lib for src/lib.rs). Tags enable filtering and reordering — see Tagged Documentation for details.
Extraction rules
For each documentation line, the extractor removes, in order:
- Leading ASCII whitespace — spaces and tabs before the slashes.
- The four slashes —
////. - Exactly one separator character — the space or tab immediately after the slashes.
Everything after the separator is preserved verbatim — no Markdown interpretation, no whitespace normalization.
The one-separator rule is what makes indentation inside your documentation possible. //// Indented code block keeps the second space, so code blocks and nested lists survive extraction intact.
Blocks
Consecutive documentation lines form a documentation block. No blank line is automatically inserted between blocks. To separate blocks visually, write an explicit //// line between them. Non-documentation source lines never appear in the output.
Source:
//// First
//// Second
let x = 5;
//// Third
Output:
First
Second
Third
Blank lines inside a block
To keep a blank line inside a block, write an empty //// line (or //// — both are equivalent):
//// First
////
//// Second
Output:
First
Second
Line endings
Line endings are preserved from the source. A file using CRLF produces CRLF output; a file using LF produces LF output.
Multi-file output
No blank line is added between files — output from multiple inputs is plain concatenation of each file's blocks, in the order given.
Markdown is your job
docextract does not interpret Markdown. It copies text verbatim. Write //// # Heading (with the space) and # Heading lands in the output. Full Markdown rendering is a non-goal — the tool is strictly an extractor.
Build Script Integration
Build scripts are the primary use case: extract macro DSL documentation into OUT_DIR on every build, then embed the generated Markdown in your crate's rustdoc output.
Setup
# Cargo.toml
[]
= "0.1"
// build.rs
use env;
Feature flags
The regex feature is enabled by default and adds regex-lite as a dependency, providing full regex pattern matching for tag filters. This is the recommended configuration for most users.
To opt into zero dependencies (no regex-lite), disable default features:
# Cargo.toml
[]
= { = "0.1", = false }
With default-features = false, pattern matching uses the lightweight hand-rolled matcher (see Pattern syntax). Tag filtering and reordering work identically — only the pattern syntax changes.
Embedding the output in your docs
Include the generated Markdown in your crate's rustdoc output via a hidden module whose doc comment is the extracted file:
Because rustdoc never sees //// comments, they cannot leak into the crate's public API docs; the extracted page is attached to the hidden module instead.
Automatic cargo:rerun-if-changed
Docextract::new() checks for OUT_DIR. When it is set (build script context), each .input() call automatically emits cargo:rerun-if-changed={path} on stdout — no manual println! needed. If your macro source changes, Cargo reruns the build script and regenerates the docs. This eliminates the classic stale-build bug of forgetting the directive.
For multi-file builds, use a loop — each .input() call emits its own directive:
use env;
Overriding auto-detection
Docextract::new() auto-detects build script context: if OUT_DIR is set, rerun_if_changed defaults to true and directives are emitted. Outside a build script (OUT_DIR unset), directives are off by default.
Use .rerun_if_changed() to override:
#
// Enable directives in a standalone tool (e.g., to embed docs during development)
new?
.rerun_if_changed
.input?
.done?;
# Ok
# }
Tagged Documentation (Filtering & Reordering)
docextract supports tagging documentation lines to enable filtering and reordering in the output. Tag directives use ////tagname syntax — a line with exactly four slashes followed by a name (no space).
Default tags
Every documentation line has a tag. The initial tag is the filename without its extension. In src/lib.rs, untagged lines get the tag lib:
// In src/lib.rs — initial tag is "lib"
//// This line has tag "lib"
////chapter1
//// This line has tag "chapter1"
Tagging lines
A bare ////tagname is a persistent tag switch — all subsequent //// lines carry this tag until another switch. The directive line itself produces no output.
A ////tagname some text is a transient tag — the content some text is emitted under tag tagname, and the previous tag resumes. Transient lines are useful for injecting glossaries, lists, or footnotes.
////chapter1
//// # Getting Started
////glossary route — a URL path bound to a handler
//// This is still under chapter1
Blocks and tags
Consecutive documentation lines under the same tag form a block. Blocks are concatenated without auto-separators — use an explicit //// line to insert a blank line.
Source:
////chapter1
//// # Getting Started
////
//// This blank line was explicit.
////glossary
//// A definition.
//// More text under glossary.
Output:
# Getting Started
This blank line was explicit.
A definition.
More text under glossary.
A persistent tag switch (////newname) ends the current block and starts a new one under the new tag. Non-documentation lines also end blocks. No blank line is ever emitted between blocks automatically — the author controls all spacing.
Filtering input
Use .input_filtered(path, pattern) to extract only lines whose tag matches a pattern:
#
Filtering output
Use .filter_output(pattern) to apply a global filter across ALL inputs:
#
Input filters and the output filter combine with AND logic — a line must pass both.
Reordering
Read the same file multiple times with different filters to reorder sections:
#
The file is read and parsed once; subsequent filtered reads operate on the cached content.
Pattern syntax
By default, patterns use full regex-lite syntax — POSIX character classes, grouping, quantifiers, and more:
[[:alpha:]]([[:alnum:]])*_bar // matches a1_bar, a123foo_bar
chapter.* // matches chapter1, chapter_intro
When built without the regex feature (--no-default-features), a lightweight hand-rolled matcher is used instead, supporting this subset:
| Pattern | Meaning |
|---|---|
.* |
Matches zero or more characters |
.? |
Matches zero or one character |
[abc] |
Character class — matches a, b, or c |
[^abc] |
Negated character class |
api |
Literal match |
The hand-rolled matcher supports no POSIX classes, no () grouping, and no +/{n,m} quantifiers. Patterns written for the reduced syntax work identically on both paths.
Transient tags in unfiltered output
Transient tags are excluded from unfiltered output — they only appear when an active filter matches their tag. This preserves backward compatibility: the transient tag directive ////glossary term definition is treated like the old silent-ignore behavior when no filters are active.
Example project
A complete, runnable example lives in examples/demo-macro — it documents a define_routes! macro and embeds the extracted Markdown into its rustdoc output.
CLI Usage
Invocation
cargo-docextract <output> [inputs...]
<output>— mandatory, must be the first argument. A file path, or-for stdout.[inputs...]— zero or more file paths. A file path, or-for stdin.
There are no options and no flags — the interface is positional only. -o, -V, --, and similar are treated as file paths. The single exception is --help, which triggers the help text from any position.
Stdin and stdout
-as the output writes to stdout.-as an input reads from stdin.- A lone
-(cargo docextract -) is shorthand forcargo docextract - -: it reads stdin and writes stdout — a pipe pass-through filter.
Help
Invoking with zero arguments or with --help anywhere prints the help text and exits 0:
cargo-docextract - extract //// documentation from Rust source files
Usage: cargo-docextract <output> [inputs...]
Arguments:
<output> Output file path, or '-' for stdout (mandatory).
Prefix with 'filter@' for output filtering,
e.g., 'chapter.*@output.md'.
[inputs...] Input file paths, or '-' for stdin (zero or more).
Prefix with 'filter@' to filter by tag pattern,
e.g., 'api@src/lib.rs'.
Examples:
cargo-docextract docs.md src/macros.rs Extract from one file
cargo-docextract - src/macros.rs Extract to stdout
cargo-docextract api@docs.md src/lib.rs Filter by 'api' tag
cargo-docextract 'chapter.*'@out.md chapter1@in.rs chapter2@in.rs
cargo-docextract - Pipe stdin → stdout
cargo-docextract --help Print this help
There are no options or flags. All arguments are positional.
Exit codes
| Code | Meaning |
|---|---|
| 0 | Success — all inputs processed, output written. Also used for help and for an empty input list. |
| 2 | I/O error — output file cannot be created, an input cannot be read, a flush fails, or no input file could be read at all. |
There is no exit code 1: with zero arguments the CLI prints help (exit 0), and with any arguments the first argument is the output — a "missing output" condition cannot arise.
Common invocations
# Unfiltered extraction (backward compatible)
cargo-docextract docs.md src/lib.rs src/macros.rs
# Output filtering — emit only lines tagged with a chapter.* pattern
cargo-docextract 'chapter.*'@chapters.md src/lib.rs
# Input filtering — only extract api-tagged lines from one file
cargo-docextract docs.md api@src/macros.rs src/lib.rs
# Reordering — read same file with different filters
cargo-docextract out.md glossary@src/macros.rs chapter1@src/macros.rs api@src/macros.rs
# Pipe stdin through, filter output to stdout
cargo-docextract 'api.*'@- -
Messages
All messages go to stderr, prefixed with cargo-docextract::
cargo-docextract: warning: input file not found: missing.rs
cargo-docextract: error: output.md: Permission denied (os error 13)
cargo-docextract: error: src/macros.rs: Permission denied (os error 13)
cargo-docextract: error: stdin: broken pipe (os error 32)
cargo-docextract: error: output: No space left on device (os error 28)
cargo-docextract: error: no input files could be read
Behavioral notes:
- File not found is a warning, not an error. Processing continues with the remaining inputs. The exit code reflects the overall outcome: 0 if at least one input was processed, 2 only if no input could be read.
- Other I/O errors abort immediately with exit code 2.
-reads stdin exactly once. If-appears twice, the second occurrence is a no-op.- An empty input list is well-defined: the output file is created empty, exit 0.
License
Licensed under the WTFPL (Do What The Fuck You Want To Public License), Version 2, December 2004.
Repository: https://seed.pipapo.org/nodes/seed.pipapo.org/rad:z446u5iBrxqbKrUrGWAuNSWkJD3PL