# `docextract` Specification
## Overview
`docextract` is a tiny Rust library and CLI for extracting documentation written using a dedicated `////` comment syntax from Rust source files.
The project intentionally **does not parse Rust syntax**. It performs a simple line-based scan of source files. Its purpose is to allow documenting declarative macro DSLs or other generated APIs independently of `rustdoc`.
The project is a **single crate** with two targets:
* **library**: `docextract` — used as a build dependency via `cargo add --build docextract`
* **binary**: `cargo-docextract` — installed via `cargo install docextract`, invoked as `cargo docextract`
The implementation should be lightweight. Zero dependencies without features; the `regex` feature (enabled by default) adds `regex-lite` as an optional dependency.
## Goals
* Extremely small implementation.
* Zero dependency on:
* syn
* quote
* proc-macro2
* clap
* anyhow
* thiserror
* regex (note: `regex-lite` is an allowed optional dependency behind the `regex` feature)
* any argument-parser crate (clap, structopt, pico-args, etc.)
The optional `regex-lite` dependency is the sole exception. When building with `--no-default-features`, the crate has zero dependencies.
* Fast startup.
* Suitable for use from `build.rs`.
* Produce deterministic output.
* Work entirely on stable Rust.
## Non-goals
* Parsing Rust syntax.
* Expanding macros.
* Understanding token trees.
* Markdown rendering.
* rustdoc integration.
* AST generation.
The tool is strictly a documentation extractor.
---
# Documentation Syntax
A line is a documentation line **only** when it begins with optional ASCII whitespace followed by `////` and then a space (0x20), tab (0x09), or end-of-line (i.e., `\n` 0x0A or `\r` 0x0D as part of CRLF). Lines that do not meet this rule are **silently ignored** — no warning is emitted.
Valid documentation lines:
```text
//// Hello
////\tHello
////
//// Hello
\t//// Hello
```
Silently ignored (no whitespace after the four slashes):
```text
////Hello
////Text
```
Silently ignored (five or more slashes):
```text
///// extra slash
////// two extra
```
After confirming the line matches the documentation-line pattern, extraction removes:
* leading ASCII whitespace (space 0x20, tab 0x09)
* the four slashes
* one separator character — the space or tab immediately following the slashes
Everything after the separator is preserved verbatim. If the line was `////` (immediate newline after slashes), `//// ` (space then newline), or `////<TAB>` (tab then newline), the result is an empty string — a verbatim blank line.
Examples:
Input
```text
//// Hello
```
Output
```text
Hello
```
Input
```text
//// ### Heading
```
Output
```text
### Heading
```
(The space between `////` and `###` is the separator — it is stripped by the extractor, leaving `### Heading`. A line written as `////### Heading` with no whitespace after the slashes is **silently ignored**, not extracted.)
Input
```text
//// Two Spaces
```
Output
```text
Two Spaces
```
The leading ASCII whitespace is stripped. The four slashes are stripped. One separator space is stripped. The remaining ` Two Spaces` (space before "Two") is preserved verbatim.
Input (tab indentation)
```text
\t//// text
```
Output
```text
text
```
Input (tab separator after slashes)
```text
////\ttext
```
Output
```text
text
```
Input (verbatim blank line from `////` alone)
```text
////
```
Output
```text
```
(`//// ` — four slashes, space, newline — produces the same empty output. Both are equivalent.)
No Markdown interpretation is performed. Whitespace inside extracted text is treated **opaquely** and passed to output verbatim — only the leading whitespace and separator are parsed; everything else is content.
Only ASCII whitespace is recognized for parsing: space (0x20) and tab (0x09). Any other whitespace characters appearing inside extracted text pass through to output unchanged.
---
# Documentation Blocks
Consecutive documentation lines form one documentation block.
Example:
```rust
//// First
//// Second
let x = 5;
//// Third
```
Produces
```text
First
Second
Third
```
No blank line is automatically inserted between documentation blocks. To separate blocks visually, write an explicit `////` (verbatim blank doc line) between them.
Blank lines inside extracted blocks are produced by explicitly writing empty `////` lines (or `//// ` lines — they are equivalent):
```rust
//// First
////
//// Second
```
Produces
```text
First
Second
```
Non-documentation source lines never appear in the output.
---
# Input
The CLI accepts zero or more input file paths.
A single hyphen `-` as an input filename reads from **stdin**. For example:
```text
cargo docextract output.md -
```
reads from stdin and writes to `output.md`. When both output and the sole input are `-` (`cargo-docextract - -`), the CLI reads stdin and writes stdout. A single `-` as the only argument (`cargo-docextract -`) means the same thing: since output-to-stdout with no inputs would otherwise produce empty output, the lone `-` implies **both** stdout output and stdin input — equivalent to `cargo-docextract - -`.
Each input file is processed independently. The output from all inputs is concatenated in the order given — there is **no separator** added between files. Multi-file output is simply the concatenation of each file's extracted documentation blocks.
An empty input list produces an empty output file. This is well-defined and useful for generated-file pipelines.
Future recursive directory walking is optional but not required for version 1.
---
# Output
Default output is stdout.
The **output path is mandatory** and comes first in the argument list (see CLI section below). To write to stdout, use `-` as the output path.
No leading or trailing blank lines are added to the output. If the source file(s) begin or end with a `////` line, those produce verbatim blank lines in the output at the corresponding position. The extractor does not add or strip blank lines at the edges of the output.
---
# CLI
Executable name:
```text
cargo-docextract
```
Invocation:
```text
cargo-docextract <output> [inputs...]
```
* `<output>` — **mandatory**, must be the first argument. A file path, or `-` for stdout.
* `[inputs...]` — zero or more file paths. A file path, or `-` for stdin.
There are **no options, no flags** (`-o`, `--`, etc. are not recognized). The interface is positional only.
**Help**: Invoking with zero arguments or `--help` prints a short help text and exits with status 0. No other flag or long-flag syntax is supported.
**Stdin**: `-` as an input filename reads from stdin. `cargo-docextract - -` reads stdin and writes stdout. A single `-` alone (`cargo-docextract -`) implies both stdout output and stdin input — equivalent to `cargo-docextract - -`, since output-to-stdout with no inputs makes no sense.
**Empty input list**: `cargo-docextract output.md` (no input files) creates an empty output file. This is well-defined behavior.
**File not found**: If an input file does not exist, print a warning to stderr, **continue** to the next file (do not abort). The exit code reflects the final outcome — success if all existing files were processed, error if none could be opened.
Other I/O errors during input processing (permission denied, read errors) cause immediate abort with exit code 2. Failure to open the output file for writing also produces exit code 2.
---
# Exit Codes
| 0 | Success |
| 2 | I/O error |
Exit code 1 is not used: with zero arguments the CLI prints help and exits 0, and with any arguments the first argument *is* the output — a "missing output argument" condition cannot arise. See [specs/cli-spec.md](specs/cli-spec.md) §6.
---
# Library API
The library exposes a single public type: the `Docextract` builder. There are
no standalone functions.
```rust
pub struct Docextract { /* ... */ }
impl Docextract {
/// Create a new builder, writing extracted documentation to `output`.
/// `"-"` means stdout. Auto-detects build-script context (OUT_DIR set)
/// and enables rerun-if-changed automatically.
pub fn new(output: impl AsRef<Path>) -> io::Result<Self>;
/// Extract documentation from a file at `path`. Processes immediately
/// (streaming). File not found → warning to stderr, returns Ok.
/// Other I/O errors → Err. Emits cargo:rerun-if-changed if enabled.
pub fn input(&mut self, path: impl AsRef<Path>) -> io::Result<&mut Self>;
/// Extract documentation from stdin. Processes immediately. No
/// cargo:rerun-if-changed emitted (stdin has no file to watch).
pub fn stdin(&mut self) -> io::Result<&mut Self>;
/// Finalize extraction: flush the output writer. Returns &mut self so the
/// call is chainable. Calling done() multiple times is safe (idempotent).
/// After done(), further .input()/.stdin() calls error with BrokenPipe.
/// Zero prior .input() calls → valid empty output file.
pub fn done(&mut self) -> io::Result<&mut Self>;
/// Control whether cargo:rerun-if-changed is emitted from .input().
/// Default is auto-detected (true in build scripts, false otherwise).
pub fn rerun_if_changed(&mut self, value: bool) -> &mut Self;
}
```
If `.done()` is not called, `Drop` flushes the writer and prints any flush
errors to stderr.
For full details, including error handling semantics, `.done()` vs `Drop`
interaction, and internal algorithms, see [specs/lib-spec.md](specs/lib-spec.md).
No global state. No configuration objects.
---
# Build Script Integration
The intended use is:
```rust
use std::env;
fn main() -> std::io::Result<()> {
let out_dir = env::var("OUT_DIR").unwrap();
docextract::Docextract::new(format!("{out_dir}/macro-docs.md"))?
.input("src/macros.rs")? // auto-detects build script → rerun-if-changed
.done()?;
Ok(())
}
```
The builder automatically detects the build script context (`OUT_DIR` is set)
and emits `cargo:rerun-if-changed=src/macros.rs` via `println!` — no manual
`println!` needed. For multi-file extraction:
```rust
let mut doc = docextract::Docextract::new(format!("{out_dir}/macro-docs.md"))?;
doc.input("src/macros.rs")?;
doc.input("src/other.rs")?;
doc.done()?;
```
Each `.input()` call emits its own `cargo:rerun-if-changed` directive.
---
# Implementation Strategy
Use `BufRead::read_line()` with `BufReader` and `BufWriter`, both configured with **32 KiB buffers**.
Rust sources are always UTF-8. If invalid UTF-8 is encountered, `read_line()` returns an `Err(io::ErrorKind::InvalidData)` which is propagated — this is a hard error, not a recoverable condition.
For every line:
1. Skip leading ASCII whitespace (space 0x20, tab 0x09).
2. Check for the prefix `////`.
3. **Validate**: the character immediately after `////` must be space (0x20), tab (0x09), or end-of-line (i.e., `\n` 0x0A or `\r` 0x0D as part of CRLF). If it is anything else (including another `/`), the line is silently ignored — no output, no warning.
4. Remove the four slashes.
5. Remove one separator character (the space or tab after the slashes, if present; nothing to remove if the line ended immediately after `////`).
6. Emit remaining text.
7. **Preserve line endings** from the input. Whatever line ending the source uses (`\n` or `\r\n`), emit that same line ending in the output. Do not normalize.
Maintain a boolean indicating whether the previous emitted line belonged to the current documentation block.
When transitioning from one block to another, no blank line is emitted — blocks are concatenated directly. Block state tracking is still needed to distinguish lines in the same block from lines in separate blocks (separated by non-documentation lines), but no separator is emitted.
No regular expressions. No parsing beyond simple prefix matching.
---
# Error Handling
Return ordinary `std::io::Error`.
Do not introduce custom error types unless genuinely necessary.
CLI errors should be human-readable and concise.
---
# Dependency Policy
**Zero dependencies by default.** The `regex` feature (enabled by default)
adds `regex-lite` as an optional dependency. `regex-lite` itself has zero
transitive dependencies (maintained by BurntSushi under the `rust-lang/regex`
monorepo, dual-licensed MIT/Apache-2.0, MSRV 1.65.0). When compiled with
`--no-default-features` (disabling the `regex` feature), the crate has
**zero dependencies** — a hand-rolled pattern matcher is used instead.
If a dependency beyond `regex-lite` is genuinely needed, it must be confirmed
by the user before being added. Be very conservative.
The following are explicitly forbidden regardless:
* syn
* quote
* proc-macro2
* clap
* structopt
* pico-args
* regex
* anyhow
* thiserror
TODO: When the tag-filter feature is implemented, update `specs/lib-spec.md` and
`specs/cli-spec.md` to reflect the design changes documented in
`specs/tag-filter-spec.md` — particularly the cache-first parsing model,
block-separator removal, and the optional `regex-lite` dependency.
---
# Testing
Include unit tests covering:
* simple extraction
* indented extraction (spaces and tabs)
* optional separator removal (space and tab after `////`)
* consecutive documentation lines forming blocks
* multiple documentation blocks concatenated without auto-separation
* empty documentation lines (`////` and `//// ` — both produce verbatim blank line)
* `////Text` — no whitespace after slashes → silently ignored
* `/////` — five (or more) slashes → silently ignored
* files with no documentation → empty output
* empty file → empty output
* documentation at EOF (including `////` as last line)
* Windows (`CRLF`) and Unix (`LF`) line endings — both preserved in output
* tabs as indentation
* trailing space after `////` (e.g., `//// ` → empty line, the space after slashes is the separator)
* long documentation line
* whitespace-only documentation line (`//// ` — three spaces: the first is the separator, the remaining two are preserved verbatim)
* invalid UTF-8 / binary input → Err(io::ErrorKind::InvalidData)
* `-` as stdin input
* input file not found → warning to stderr, continue
* zero input arguments → empty output file
* `--help` → prints help, exits 0
* zero arguments → prints help, exits 0
* "missing output argument" is unreachable — the first argument is always the output; zero arguments prints help, exits 0
---
# Future Extensions (Not Part of Version 1)
Potential future additions include:
* recursive directory traversal
* `--extension`
* `--check`
* `--fail-on-empty`
* `--heading-from-file`
* `--strip-indent`
* configurable documentation prefixes
* JSON output
* Markdown table of contents generation
These are explicitly out of scope for the initial implementation.
---
# Packaging
**Cargo.toml metadata**: `license`, `repository`, and `description` fields will be added before publishing. A `README.md` will be written separately.
(End of file)