docextract 0.4.0

documentation extractor
Documentation
  • Coverage
  • 100%
    10 out of 10 items documented2 out of 10 items with examples
  • Size
  • Source code size: 249.5 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 568.9 kB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 2s Average build duration of successful builds.
  • all releases: 3s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • Repository
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • cehteh

docextract

Extract //// documentation from Rust source files.

docextract is a tiny Rust library and CLI for pulling dedicated //// comments out of Rust source files into a standalone Markdown document. It is the missing piece for documenting declarative macro DSLs and generated APIs — things rustdoc cannot see — and it does this with a strict line-scanning approach: no Rust parsing, no token trees, no macro expansion, and zero dependencies by default (an optional regex feature adds regex-lite for advanced pattern matching). A build.rs step (or the CLI) extracts these comments into a standalone Markdown file that ships with your crate or lives alongside your documentation.

Note: This crate was developed with AI assistance.


Documentation Syntax

What counts as a documentation line

A line is a documentation line when it starts with optional ASCII whitespace (spaces or tabs), followed by exactly four slashes ////, followed by a space (0x20), a tab (0x09), or the end of the line.

When //// appears at the end of a source line (preceded by a space or tab), the line becomes a code-capturing line. The code on the left is captured, and {{...}} placeholders in the text on the right are substituted with the captured code.

Valid documentation lines:

//// Hello                space separator
////   Hello              extra spaces are content
////	Hello             tab separator (tab after the slashes)
////                      empty doc line → verbatim blank line
    //// Hello            indented with spaces
	//// Hello            indented with a tab
let x = 42;  //// {{}}    code-capturing line — {{}} substitutes the code

Tag directives

When a //// line has no separator (the character immediately after the slashes is not a space, tab, or newline), it is a tag directive:

////chapter1              persistent tag switch — all following doc lines carry this tag
////glossary definition   transient tag — this line only, then tag reverts

The initial tag is the filename without its extension (e.g., lib for src/lib.rs). Tags enable filtering and reordering — see Tagged Documentation for details.

Extraction rules

For each documentation line, the extractor removes, in order:

  1. Leading ASCII whitespace — spaces and tabs before the slashes.
  2. The four slashes — ////.
  3. Exactly one separator character — the space or tab immediately after the slashes.

Everything after the separator is preserved verbatim — no Markdown interpretation, no whitespace normalization.

The one-separator rule is what makes indentation inside your documentation possible. //// Indented code block keeps the second space, so code blocks and nested lists survive extraction intact.

{{...}} in regular doc lines is literal. Placeholder substitution only happens on code-capturing lines (where //// is at the end). A regular //// {{}} doc line emits {{}} verbatim. To use placeholders, write code before the ////:

//// {{}}              → emitted as "{{}}" (verbatim — regular doc line)
let x = 42;  //// {{}}  → emitted as "let x = 42;" (code-capturing line — placeholder resolved)

Blocks

Consecutive documentation lines form a documentation block. No blank line is automatically inserted between blocks. To separate blocks visually, write an explicit //// line between them. Non-documentation source lines never appear in the output, except for code-capturing lines (lines ending with ////) which emit the resolved template text.

Source:

//// First
//// Second

let x = 5;

//// Third

Output:

First
Second
Third

Blank lines inside a block

To keep a blank line inside a block, write an empty //// line (or //// — both are equivalent):

//// First
////
//// Second

Output:

First

Second

Line endings

Line endings are preserved from the source. A file using CRLF produces CRLF output; a file using LF produces LF output.

Multi-file output

No blank line is added between files — output from multiple inputs is plain concatenation of each file's blocks, in the order given.

Markdown is your job

docextract does not interpret Markdown. It copies text verbatim. Write //// # Heading (with the space) and # Heading lands in the output. Full Markdown rendering is a non-goal — the tool is strictly an extractor.

Code-Lifting Syntax

Non-documentation source lines ending with //// capture the code on the left side for placeholder substitution in the doc text on the right side.

Placeholders:

  • {{}} — substitutes the captured code with leading and trailing whitespace stripped
  • {{N}} — removes the first N UTF-8 characters, substitutes the remainder
  • {{/regex/}} — finds regex in the code, substitutes the match (extract mode)
  • {{/regex/subst/}} — matches regex and outputs only the expanded substitution (using $1, $2 backreferences). Non-matched text is discarded. No match → empty output.

Any ASCII punctuation can be the delimiter: {{!regex!subst!}}

Examples:

let x = 42;       //// {{}}
// let x = 42;

let x = 42;       //// The answer is {{}}.
// The answer is let x = 42;

    let foo=bar;  //// {{8}}
// foo=bar;

let x = 42;       //// {{/let ([[:alnum:]]+) = (.+);/$1 is $2/}}
// x is 42

Code-capturing lines produce regular persistent documentation entries. They carry the current tag and respect all filtering rules. The {{}} and {{N}} forms work without any feature flags; the sed-like forms ({{XregexX}} / {{XregexXsubstX}}) require the regex feature (enabled by default).

Capture zone continuation

Lines ending with //// establish a capture zone at the indentation of the captured code. Within this zone, //// lines at deeper indentation are treated as code-capturing suffixes (not doc prefixes) — the leading whitespace becomes empty captured code, and the template resolves normally:

    code;  //// {{}}          ← establishes capture zone at indent 4
           //// {{4}} suffix  ← deeper indent — continuation, {{4}} on empty → "", appends " suffix"
    more;  //// {{}}          ← same indent — regular code-capturing line, zone stays open
//// end                      ← indent 0 ≤ 4 — zone resets, regular doc line

Produces:

code;
 suffix
more;
end

The zone ends when a line at ≤ the zone's indentation is encountered (non-code-capturing lines, tag directives, or //// lines at the zone indent or shallower).


Build Script Integration

Build scripts are the primary use case: extract macro DSL documentation into OUT_DIR on every build, then embed the generated Markdown in your crate's rustdoc output.

Setup

# Cargo.toml
[build-dependencies]
docextract = "0.1"
// build.rs
use std::env;

fn main() -> std::io::Result<()> {
    let out_dir = env::var("OUT_DIR").unwrap();

    docextract::Docextract::new(format!("{out_dir}/macro-docs.md"))?
        .input("src/macros.rs")?   // auto-detects build script context
        .done()?;

    Ok(())
}

Feature flags

The regex feature is enabled by default and adds regex-lite as a dependency, providing full regex pattern matching for tag filters. This is the recommended configuration for most users.

To opt into zero dependencies (no regex-lite), disable default features:

# Cargo.toml
[build-dependencies]
docextract = { version = "0.1", default-features = false }

With default-features = false, pattern matching uses the lightweight hand-rolled matcher (see Pattern syntax). Tag filtering and reordering work identically — only the pattern syntax changes.

Embedding the output in your docs

Include the generated Markdown in your crate's rustdoc output via a hidden module whose doc comment is the extracted file:

#[doc = include_str!(concat!(env!("OUT_DIR"), "/macro-docs.md"))]
pub mod _extracted_docs {}

Because rustdoc never sees //// comments, they cannot leak into the crate's public API docs; the extracted page is attached to the hidden module instead.

Automatic cargo:rerun-if-changed

Docextract::new() checks for OUT_DIR. When it is set (build script context), each .input() call automatically emits cargo:rerun-if-changed={path} on stdout — no manual println! needed. If your macro source changes, Cargo reruns the build script and regenerates the docs. This eliminates the classic stale-build bug of forgetting the directive.

For multi-file builds, use a loop — each .input() call emits its own directive:

use std::env;

fn main() -> std::io::Result<()> {
    let out_dir = env::var("OUT_DIR").unwrap();
    let files = ["src/macros.rs", "src/other.rs"];

    let mut extractor = docextract::Docextract::new(format!("{out_dir}/macro-docs.md"))?;
    for file in files {
        extractor.input(file)?;
    }
    extractor.done()?;

    Ok(())
}

Overriding auto-detection

Docextract::new() auto-detects build script context: if OUT_DIR is set, rerun_if_changed defaults to true and directives are emitted. Outside a build script (OUT_DIR unset), directives are off by default.

Use .rerun_if_changed() to override:

# fn main() -> std::io::Result<()> {
# let out_dir = "/tmp/out";
// Disable directives inside a build script
docextract::Docextract::new(format!("{out_dir}/macro-docs.md"))?
    .rerun_if_changed(false)
    .input("src/macros.rs")?
    .done()?;
# Ok(())
# }
# fn main() -> std::io::Result<()> {
// Enable directives in a standalone tool
docextract::Docextract::new("docs.md")?
    .rerun_if_changed(true)
    .input("src/docs.rs")?
    .done()?;
# Ok(())
# }

Tagged Documentation (Filtering & Reordering)

docextract supports tagging documentation lines to enable filtering and reordering in the output. Tag directives use ////tagname syntax — a line with exactly four slashes followed by a name (no space).

Default tags

Every documentation line has a tag. The initial tag is the filename without its extension. In src/lib.rs, untagged lines get the tag lib:

// In src/lib.rs — initial tag is "lib"
//// This line has tag "lib"
////chapter1
//// This line has tag "chapter1"

Tagging lines

A bare ////tagname is a persistent tag switch — all subsequent //// lines carry this tag until another switch. The directive line itself produces no output.

A ////tagname some text is a transient tag — the content some text is emitted under tag tagname, and the previous tag resumes. Transient lines are useful for injecting glossaries, lists, or footnotes.

////chapter1
//// # Getting Started
////glossary route — a URL path bound to a handler
//// This is still under chapter1

Blocks and tags

Consecutive documentation lines under the same tag form a block. Blocks are concatenated without auto-separators — use an explicit //// line to insert a blank line.

Source:
////chapter1
//// # Getting Started
////
//// This blank line was explicit.
////glossary
//// A definition.
//// More text under glossary.

Output:
# Getting Started

This blank line was explicit.
A definition.
More text under glossary.

A persistent tag switch (////newname) ends the current block and starts a new one under the new tag. Non-documentation lines also end blocks. No blank line is ever emitted between blocks automatically — the author controls all spacing.

Filtering input

Use .input_filtered(path, pattern) to extract only lines whose tag matches a pattern:

# fn main() -> std::io::Result<()> {
// Extract only lines tagged "api"
docextract::Docextract::new("api-docs.md")?
    .input_filtered("src/macros.rs", "api")?
    .done()?;
# Ok(())
# }

Filtering output

Use .filter_output(pattern) to apply a global filter across ALL inputs:

# fn main() -> std::io::Result<()> {
// Extract chapter.* lines from all sources
docextract::Docextract::new("chapters.md")?
    .filter_output(Some("chapter.*"))?
    .input("src/intro.rs")?
    .input("src/advanced.rs")?
    .done()?;
# Ok(())
# }

Input filters and the output filter combine with AND logic — a line must pass both.

Reordering

Read the same file multiple times with different filters to reorder sections:

# fn main() -> std::io::Result<()> {
docextract::Docextract::new("output.md")?
    .input_filtered("src/docs.rs", "chapter3")?   // last chapter first
    .input_filtered("src/docs.rs", "chapter2")?
    .input_filtered("src/docs.rs", "chapter1")?
    .input_filtered("src/docs.rs", "glossary")?   // glossary last
    .done()?;
# Ok(())
# }

The file is read and parsed once; subsequent filtered reads operate on the cached content.

Pattern syntax

By default, patterns use full regex-lite syntax — POSIX character classes, grouping, quantifiers, and more:

[[:alpha:]]([[:alnum:]])*_bar    // matches a1_bar, a123foo_bar
chapter.*                          // matches chapter1, chapter_intro

When built without the regex feature (--no-default-features), a lightweight hand-rolled matcher is used instead, supporting this subset:

Pattern Meaning
.* Matches zero or more characters
.? Matches zero or one character
[abc] Character class — matches a, b, or c
[^abc] Negated character class
api Literal match

The hand-rolled matcher supports no POSIX classes, no () grouping, and no +/{n,m} quantifiers. Patterns written for the reduced syntax work identically on both paths.

Transient tags in unfiltered output

Transient tags are excluded from unfiltered output — they only appear when an active filter matches their tag. This preserves backward compatibility: the transient tag directive ////glossary term definition is treated like the old silent-ignore behavior when no filters are active.

Example project

The demo-macro example demonstrates the full docextract pipeline:

  1. A Rust crate uses //// comment syntax to document a declarative macro DSL
  2. build.rs extracts those //// comments into $OUT_DIR/macro-docs.md
  3. The extracted docs are re-integrated into cargo doc output via #[doc = include_str!(...)]
  4. The define_routes! macro and its generated dispatch function are exercised by unit tests

How It Works

Source documentation (src/lib.rs): The crate uses //// comment syntax throughout to document the define_routes! macro syntax, generated code patterns, usage examples, and tests. These comments are invisible to rustdoc — keeping macro DSL documentation separate from the public API surface.

Build script (build.rs):

# fn main() -> std::io::Result<()> {
# let out_dir = std::path::PathBuf::from("/tmp/out");
// In build.rs:
docextract::Docextract::new(&out_dir.join("macro-docs.md"))?
    .input("src/lib.rs")?
    .done()?;
# Ok(())
# }

On every build, build.rs scans src/lib.rs for //// doc lines and writes the extracted content to $OUT_DIR/macro-docs.md. The builder automatically emits cargo:rerun-if-changed directives.

Rustdoc integration (src/lib.rs):

#[doc = include_str!(concat!(env!("OUT_DIR"), "/macro-docs.md"))]
mod _extracted_docs {}

This embeds the extracted documentation into a dedicated module page in cargo doc output.

Building and Testing

cargo build                      # triggers build.rs to extract docs
cargo test -p demo-macro         # runs tests
cargo doc --open                 # view extracted docs under _extracted_docs

CLI Usage

Invocation

cargo-docextract <output> [inputs...]
  • <output> — mandatory, must be the first argument. A file path, or - for stdout.
  • [inputs...]. Zero or more file paths. A file path, or - for stdin.

There are no options and no flags, the interface is positional only. -o, -V, --, and similar are treated as file paths. The single exception is --help, which triggers the help text from any position.

Stdin and stdout

  • - as the output writes to stdout.
  • - as an input reads from stdin.
  • A lone - (cargo docextract -) is shorthand for cargo docextract - -: it reads stdin and writes stdout — a pipe pass-through filter.

Help

Invoking with zero arguments or with --help anywhere prints the help text and exits 0:

cargo-docextract - extract //// documentation from Rust source files

Usage: cargo-docextract <output> [inputs...]

Arguments:
  <output>       Output file path, or '-' for stdout (mandatory).
                 Prefix with 'filter@' for output filtering,
                 e.g., 'chapter.*@output.md'.
  [inputs...]    Input file paths, or '-' for stdin (zero or more).
                 Prefix with 'filter@' to filter by tag pattern,
                 e.g., 'api@src/lib.rs'.

Examples:
  cargo-docextract docs.md src/macros.rs         Extract from one file
  cargo-docextract - src/macros.rs               Extract to stdout
  cargo-docextract api@docs.md src/lib.rs        Filter by 'api' tag
  cargo-docextract 'chapter.*'@out.md chapter1@in.rs chapter2@in.rs
  cargo-docextract -                             Pipe stdin → stdout
  cargo-docextract --help                        Print this help

There are no options or flags. All arguments are positional.

Exit codes

Code Meaning
0 Success — all inputs processed, output written. Also used for help and for an empty input list.
2 I/O error — output file cannot be created, an input cannot be read, a flush fails, or no input file could be read at all.

There is no exit code 1: with zero arguments the CLI prints help (exit 0), and with any arguments the first argument is the output — a "missing output" condition cannot arise.

Common invocations

# Unfiltered extraction (backward compatible)
cargo-docextract docs.md src/lib.rs src/macros.rs

# Output filtering — emit only lines tagged with a chapter.* pattern
cargo-docextract 'chapter.*'@chapters.md src/lib.rs

# Input filtering — only extract api-tagged lines from one file
cargo-docextract docs.md api@src/macros.rs src/lib.rs

# Reordering — read same file with different filters
cargo-docextract out.md glossary@src/macros.rs chapter1@src/macros.rs api@src/macros.rs

# Pipe stdin through, filter output to stdout
cargo-docextract 'api.*'@- -

Messages

All messages go to stderr, prefixed with cargo-docextract::

cargo-docextract: warning: input file not found: missing.rs
cargo-docextract: error: output.md: Permission denied (os error 13)
cargo-docextract: error: src/macros.rs: Permission denied (os error 13)
cargo-docextract: error: stdin: broken pipe (os error 32)
cargo-docextract: error: output: No space left on device (os error 28)
cargo-docextract: error: no input files could be read

Behavioral notes:

  • File not found is a warning, not an error. Processing continues with the remaining inputs. The exit code reflects the overall outcome: 0 if at least one input was processed, 2 only if no input could be read.
  • Other I/O errors abort immediately with exit code 2.
  • - reads stdin exactly once. If - appears twice, the second occurrence is a no-op.
  • An empty input list is well-defined: the output file is created empty, exit 0.

License

Licensed under the WTFPL (Do What The Fuck You Want To Public License), Version 2, December 2004.

Repository: https://seed.pipapo.org/nodes/seed.pipapo.org/rad:z446u5iBrxqbKrUrGWAuNSWkJD3PL