docattest for Rust
docattest for Rust projects: documentation that CI can fail. Your Markdown pages carry examples that run, facts that are compared with values in your code, tables whose rows are tests, and citations of tests you already have. The examples in your doc comments are checked the same way.
This package is what a Rust user installs. It provides two commands:
docattest, the command you run. The docattest core is written in Rust, so this command is built with the core inside it: it tells the core how to start the Rust adapter and what command runs docattest in your project, and runs it.docattest-adapter-rust, the adapter: the program that answers everything that needs Rust. You do not run it yourself. It speaks the adapter protocol, version 1.
Your project needs cargo on the PATH, as it does to be built: examples and facts are
compiled by it. Node, Python and Go are never needed.
Install
That builds both commands, the core among them (the crate docattest-core), and puts them
side by side in cargo's bin directory. Nothing is downloaded when docattest runs. From a
checkout of the repository, cargo install --path adapters/rust does the same.
Another core runs in place of the one built in when DOCATTEST_CORE names it.
That core is asked for its version (--version --json) and refused when it speaks another
protocol than this package.
DOCATTEST_TRACE=1 says which core runs. A bare core, run without this command, finds the
adapter as docattest-adapter-rust on the PATH, or through DOCATTEST_ADAPTER_RUST.
That variable holds a JSON array (["/path/to/docattest-adapter-rust"]) or a command line
split on spaces.
DOCATTEST_CARGO names the cargo the adapter runs, when it is not the one on the PATH:
$ DOCATTEST_CARGO=/no/such/cargo docattest check docs/pricing.md | head -2
docs/pricing.md
✗ L5 example: cargo could not read the project: /no/such/cargo could not be run: No such file or directory (os error 2) (block line 1)
docattest hooks install, hooks install --git and agents install write down a command
that runs docattest: docattest when that is what a shell finds on the PATH, else this
command by its full path.
To have them write something else, set DOCATTEST_COMMAND.
Configure
docattest.config.json at the project root:
surfacesname what pages are allowed to observe. A surface is one module of a library of your cargo workspace. An entry is- a module path,
"shop"or"shop::shipping": the crate's root, or a module below it; - a path,
"."(the package there, by its root module) or"./src/shipping.rs"(the module that file is); - both,
{ "import": "shop::shipping", "path": "src/shipping.rs" }.
- a module path,
- What a surface exports is every
pubitem its module declares, except modules, and what it re-exports withpub use(a glob's by the exports of the module it names).uncoveredlists the exports no page attests. A crate whose root only declares modules exports nothing at its root: name the modules as surfaces. sourcesare the Rust files whose doc-comment examples are checked (see below).setupis one Rust file with the runners of cases tables and what facts may use by name.
A page
A discount is a whole percentage taken off a price.
```rust check
use shop::{apply_discount, Money};
let price = apply_discount(Money::eur(1000), 25).unwrap();
assert_eq!(price.to_string(), "7.50 EUR");
```
Prices are in `EUR` unless told otherwise. <!-- fact: DEFAULT_CURRENCY -->
Tax is known for these regions: <!-- fact: TAX_RATES -->
- --
Discounts above the maximum are refused. <!-- test: src/lib.rs > "tests" > "rejects_big_discounts" -->
When a page and the code disagree, docattest check lists each proof that does not hold,
by its line, with what went wrong, and exits with 1. Here it is on a page of this package's
test fixture that is wrong on purpose:
$ docattest check docs/broken.md
docs/broken.md
✗ L5 example: panicked: assertion `left == right` failed: expected Ok(Money { cents: 700, currency: "EUR" }), got Ok(Money { cents: 750, currency: "EUR" }) (block line 3)
✗ L11 example: unresolved import `shop::aply_discount` (block line 1)
✗ L15 example: panicked: called `Result::unwrap()` on an `Err` value: "no carrier serves \"XX\"" (block line 1)
✗ L19 example: timed out after 3000 ms
✗ L23 example: ran to its end, and says it panics
! L27 example: example at L27: unknown tag "chekc" (known: check, compile, network, should_panic; add renderer tags to "ignoreTags")
✗ L31 fact: expected 60, got 50 from MAX_DISCOUNT
✗ L33 fact: expected 2 items from the page, got 3 from TAX_RATES; missing: DE, FR; unexpected: BE
! L37 fact: fact `round(50)`: round is not exported by surface "default" (cannot find function `round` in this scope)
✗ L42 cases: docs/broken.md:42 wrong on purpose: expected 800, got 750
✗ L43 cases: docs/broken.md:43 overflows: panicked: attempt to multiply with overflow
✗ L44 cases: docs/broken.md:44 too much: The discount is outside the allowed range
! L46 cases: cases runner "nope" at L46 is not in CASES of docs/docattest_setup.rs (known: delivery, discount)
! L51 test: src/lib.rs declares no test "test_that_was_renamed"; tests there: "tests" > "adds_tax", "tests" > "rejects_big_discounts"
! L53 test: "tests" in src/lib.rs is a module, not a test; tests there: "tests" > "adds_tax", "tests" > "rejects_big_discounts"
results: 0 pass, 10 fail, 5 error, 0 skipped
...
FAILED
[exit 1]
What each attestation means in Rust
Examples: rust check, rust compile
Blocks tagged rust or rs are the adapter's.
A block is read as rustdoc reads a doc test, and compiled as a program of its own that
depends on your crates.
-
What a block is. A block that declares
fn mainis a whole program. Any other is the body ofmain: Rust lets a function holdusedeclarations and items, so a snippet needs nothing more. A line that begins#is part of the program and hidden by renderers (rustdoc, mdBook). -
checkbuilds the block and runs it. The proof is an assertion (the macros and methods that count are listed below). A failedassert_eq!reports both sides, and every panic the line of the block it came from. A block that holds none of these runs, but is counted as weak. -
check should_panicpasses when the block panics, and fails when it runs to its end. -
compilebuilds the block and never runs it. The diagnostics are the compiler's own errors, at the lines of the block. Warnings are not reported: a snippet is not held to a crate's lints, as a doc test is not. -
Where it is built. In a crate the adapter generates in your target directory (
target/docattest), never inside your sources. That crate depends on the libraries of your workspace by path and on what they depend on, dev-dependencies included, so a block may use what a doc test may use; it carries your workspace's[patch]tables and starts from yourCargo.lock. What cargo built there is still there the next time: the first run of a project compiles its dependencies once more, for this crate, and later runs build only what changed. Deletingtarget/docattestcosts nothing but that time. In the fixture:$ rm -rf target/docattest $ docattest check --quiet docs/pricing.md ... $ ls target/docattest/crate Cargo.lock Cargo.toml src -
Isolation. Every block is its own program and its own process, started in the project root. Blocks share nothing. This guide's project root is
adapters/rust:assert!; assert!; -
Time. A block that outlives
runtime.timeoutMsis killed with everything it started.runtime.workersis the number of blocks that run at once. Building is not counted: the blocks of a run are built together, once, before any of them runs. -
Output. What a block prints is dropped; it never reaches the report. The block of the fixture's pricing page at L35 prints a price and asserts nothing:
$ docattest check docs/pricing.md docs/pricing.md ✓ L5 example: all assertions held ✓ L14 example: all assertions held ✓ L25 example: compiles ✓ L35 example: ran to its end, but asserts nothing (weak) ✓ L43 example: all assertions held ... ok -
Network. A block tagged
check networkruns only withDOCATTEST_NETWORK=1, and is skipped otherwise. The network is not taken away from other blocks: Rust has no place to refuse it from inside a program, and this adapter does not sandbox what it runs.$ docattest check docs/shipping.md | grep network - L24 example: network example skipped: the network is off (set DOCATTEST_NETWORK=1 to run it)
A block asserts when it calls one of these macros:
assert!assert_eq!assert_ne!assert_matches!debug_assert!debug_assert_eq!debug_assert_ne!panic!unreachable!
or one of these methods, which end the program on a value they do not expect:
unwrapexpectunwrap_errexpect_err
Examples in doc comments
Every Rust block in a doc comment (///, //!) of a sources file is an example, as
cargo test --doc takes it: a block with no info string, or with only these words:
rustno_runshould_panicedition2015edition2018edition2021edition2024
Any other word leaves the block alone, ignore and compile_fail among them. no_run is
compiled and not run, and should_panic must panic. The claim an example defends is the
prose of its doc comment, and it attests the item it documents (a method's attests its
type).
/// Takes a percentage off a price, rounding half up to a whole cent.
///
/// ```
/// use shop::{apply_discount, Money};
///
/// assert_eq!(apply_discount(Money::eur(999), 50), Ok(Money::eur(500)));
/// ```
That is the fixture's apply_discount, and its total has a no_run example. Both are
checked, and each is listed under the item it documents, with its doc comment's sentence:
$ docattest check src/lib.rs
src/lib.rs
✓ L46 example: all assertions held
✓ L60 example: compiles
...
$ docattest where total | grep src/
src/lib.rs:60 example Adds up a basket and the tax of a region.
Facts
A fact expression is a Rust expression over the surface's exports, by their own names:
A discount is at most `50` percent. <!-- fact: MAX_DISCOUNT -->
The carriers: <!-- fact @shipping: CARRIERS -->
It may use the surface's exports, the pub items of the setup file, snapshot("name"),
manifest("path"), and anything of std by its path. A name the surface does not export is an error
(round is not exported by surface "default"), which is what keeps a page honest about
what a user can see.
The page's literal is compared with the value as Rust displays it: Display when the type
has it, else Debug. So true, 3, 7.50 EUR for a type that prints itself so, and
Some(3) or ["a", "b"] for one that has only Debug. A list on the page is compared
with the items of anything a reference to which can be iterated (an array, a slice, a
Vec, a set), each shown the same way. A map's items are its keys (HashMap, sorted, and
BTreeMap), and so are those of a list of pairs, which is how a constant map is usually
written: <!-- fact: TAX_RATES --> above a list checks the keys of
pub const TAX_RATES: [(&str, u32); 3].
snapshot("name") is the snapshot's data as a small JSON value: index it with a key or a
position, iterate an array, and to_string() a member:
snapshot("regions").iter().map(|r| r["code"].to_string()).collect::<Vec<_>>().
manifest("Cargo.toml") is the data of a project file, read and parsed by the core (JSON,
TOML and go.mod; see manifests), as the same JSON value:
manifest("Cargo.toml")["package"]["rust-version"], whose strings are text, as a String
is. The path is
relative to the project root and written out in the fact: a path built at run time is
never read.
Cases tables and named facts: the setup file
// docs/docattest_setup.rs
use ;
use HashMap;
type Row = ;
/// The runners of cases tables, by name. A runner is called once per row with the row's
/// cells by column; it returns `Ok` when the row holds, else why it does not.
pub const CASES: & = &;
/// (Optional.) The columns each runner's table must have.
pub const COLUMNS: & = &;
/// Something a fact may use: <!-- fact: error_messages() -->
The setup file is no part of your crates. docattest compiles it, as a module of a program
of its own that depends on them, so it imports your crate by its name, as a page does, and
cargo build never sees it.
A runner that panics fails its row, and the next row still runs:
$ docattest check docs/broken.md | grep ' cases: '
✗ L42 cases: docs/broken.md:42 wrong on purpose: expected 800, got 750
✗ L43 cases: docs/broken.md:43 overflows: panicked: attempt to multiply with overflow
✗ L44 cases: docs/broken.md:44 too much: The discount is outside the allowed range
! L46 cases: cases runner "nope" at L46 is not in CASES of docs/docattest_setup.rs (known: delivery, discount)
Citations
<!-- test: src/lib.rs > "tests" > "rejects_big_discounts" --> ties a sentence to a test.
A test is a function with a #[test] attribute (or one that ends in test, such as
#[tokio::test]), titled by its name, with the modules written around it named first,
outermost first. Citing a module is an error that says so
("tests" in src/lib.rs is a module, not a test).
The file is read, never run.
What is never your project's own
The adapter tells the core to leave every target directory alone. Hidden directories are
left out by the core itself.
What is read, and what is asked of the compiler
Exports, the index (where, uncovered) and impact are read from your source as it is
written, without compiling it. That is fast, and it is exact for a module whose items are
all written out.
A module's source does not say everything it exports when it calls a macro where its
items are (thread_local!, include!, a macro of your own), or re-exports whole
something its source does not lead to (pub use Kind::* of an enum, a module of another
crate). For a surface that is such a module, the adapter asks the compiler, through
rustdoc's JSON output, and lists what the compiler lists. This happens by itself, and
only for such a surface. The listing is built beside the examples, in
target/docattest/doc, and costs one cargo check of your dependencies the first time.
rustdoc's JSON is not stable yet, so the adapter sets RUSTC_BOOTSTRAP to the name of
your crate for that one command. When the compiler cannot answer (your crate does not
compile, or a later Rust writes a format this adapter does not read), check says so as
a warning and the surface keeps the exports its source declares.
Limits
-
impacthas no type information, and errs towards naming too much. It cuts a crate into the names its modules declare, with what animplblock holds as part of its type; a path stands for any item of that name, a method is found through its type, and a trait is not followed to what implements it. -
What a macro writes has no declaration to compare, so a change to a macro call written where a module's items are affects everything that module declares. An export a macro wrote that a change removes is not named: the earlier revision is not compiled.
-
A
pubitem of a private module counts as an export of that module, whether or not anything outside can reach it. What a glob re-export of another crate brings is not listed. -
A surface is a module of a crate of your workspace, not of a dependency.
-
auditandinventorysee a surface's exports that are values, and not what lies below them (the fields of a struct, the values of a map), and on the default surface what the setup module'spubfunctions that take no arguments return. In the fixture,Moneyis a struct andratestakes the regions:$ docattest inventory DEFAULT_CURRENCY → EUR MAX_DISCOUNT → 50 TAX_RATES → [NL, DE, FR] ERROR_CODES → [out_of_stock, bad_discount, no_carrier] error_messages() → [The item is not available, The discount is outside the allowed range, No carrier serves the region] @shipping CARRIERS → [post, courier, drone] inventory: 6 values -
The network is not refused (see above).
-
Two docattest runs on one project at the same time share
target/docattest: cargo makes one build wait for the other, but they may rewrite each other's generated files. -
This adapter has not been run on Windows.
Working on this package
CONTRIBUTING.md has how the crate is laid out, and how to test, lint and format it.