rsconstruct 0.9.167

Rust based fast build system
# Cache System

RSConstruct uses a content-addressed cache to enable fast incremental builds. This page describes the cache architecture, storage format, and rebuild logic.

## Overview

The cache lives in `.rsconstruct/` and consists of:

- `objects/` — content-addressed blob store, sharded by the first two hex
  characters of the content hash
- `descriptors/` — cache descriptors (marker/blob/tree JSON), sharded the
  same way but addressed by the descriptor key rather than by content
- `db.redb` — redb database holding the `processor_configs` table used for
  config-change diffing and the `product_last_tree` table used to unlink
  stale outputs (see [Stale outputs before a rebuild](#stale-outputs-before-a-rebuild))
- `deps.redb` — source file dependency cache (see [Dependency Caching](dependency-caching.md))
- `mtime.redb` — mtime→checksum pre-check cache that lets unchanged files
  skip re-hashing
- `webcache.redb` — HTTP request cache for processors that fetch remote
  resources

## Data model

The object store contains three kinds of objects, inspired by git:

### Blobs

A blob is a file's raw content, addressed by its SHA-256 content hash. Blobs are optionally zstd-compressed and made read-only to prevent corruption when restored via hardlinks.

Blobs are stored content-addressed — two products producing identical output share the same blob. This enables deduplication and hardlink-based restoration.

### Why blobs don't store output paths

A blob is pure content — it has no knowledge of where it will be restored. This is critical for two reasons:

1. **Rename survival.** If you rename `foo.md` to `bar.md` without changing its content, the cache key (which is content-addressed) is the same. The blob is reused and restored to the new output path (`bar.txt` instead of `foo.txt`). If the blob stored its output path, this wouldn't work.

2. **Deduplication across trees.** Multiple tree entries can point to the same blob under different paths. For example, if two files in an explicit stanza's output directory have identical content, they share the same blob object in the store. The tree records the path; the blob just holds the content.

### Trees

A tree is a serialized list of `(path, mode, blob_checksum)` entries describing a set of output files. Trees are stored as descriptors under `descriptors/`, addressed by the descriptor key (not by content hash). A tree maps relative file paths to content-addressed blobs. Multiple trees can point to the same blobs — deduplication happens at the blob level.

### Markers

A marker is a minimal descriptor indicating that a check passed. Markers are stored under `descriptors/`, addressed by the descriptor key, and reference no blobs.

### Cache entries

A cache entry is a small JSON descriptor stored under `.rsconstruct/descriptors/` at the path derived from the descriptor key. It contains:

```json
{"type": "blob", "checksum": "abc123...", "mode": 493}
```

Note: the blob descriptor has no path — the product knows where its output goes.

Or:

```json
{"type": "tree", "entries": [{"path": "dir/file.txt", "checksum": "def456...", "mode": 493}]}
```

or:

```json
{"type": "marker"}
```

The actual file content lives in separate content-addressed blob objects. The cache entry is just a pointer (for a single output file) or a manifest (for a product with several outputs or `output_dirs`).

### On-disk layout

```text
.rsconstruct/objects/
  a1/b2c3d4...    # blob (raw file content), addressed by content hash
  cd/ef5678...    # another blob (".zst" suffix when compressed)
.rsconstruct/descriptors/
  ff/0011aa...    # descriptor (JSON), addressed by descriptor key
```

Descriptors and blobs live in separate directories with the same two-hex-char
sharding. Keeping them separate is what lets `cache trim` delete unreferenced
blobs by scanning the descriptors.

## Cache keys

The descriptor key identifies a product build. It is computed as
(`CacheKey::descriptor_key` in `src/cache_key.rs`):

```text
hash(processor_name, processor_version, cache_key_digest, input_content_hash)
```

Where:
- `processor_name` — the processor type (e.g., `pandoc`, `ruff`)
- `processor_version` — the processor's cache version from the plugin
  registry; bumping it invalidates every entry the processor ever produced
  (see [Processor versioning](processor-versioning.md))
- `cache_key_digest` — a digest of every piece of **non-input** state that
  affects the output (below)
- `input_content_hash` — combined SHA-256 hash of all input file contents

The key is **content-addressed**: it depends on what the inputs contain, not what they're named. Renaming a file without changing its content produces the same cache key.

### Cache key components

`CacheKey` is the single owner of key composition. Key material used to be
assembled in five uncoordinated places, all splicing into one opaque
`config_hash: Option<String>` with three different separator conventions —
so a component that silently failed to be mixed in was invisible, and a
changed key could not be attributed to a cause. Contributors now append a
named, typed component and one function folds them into the digest:

| Component | Contributed by | Covers |
|---|---|---|
| `config` | discovery, via `output_config_hash` over the processor's `checksum_fields()` | compiler flags, args, output declarations |
| `variant` | `Product::with_variant` | multiple products from one input (e.g. one per output format) |
| `analyzer` | `Product::extend_config_hash` | non-content state an analyzer resolves, e.g. the sorted set of paths matching a glob |
| `tool` | `apply_tool_version_hashes`, gated on [`hash_tool_versions`](../configuration.md#build) | identity of the external tools the processor invokes |
| `python_env` | `apply_python_env`, gated on [`hash_python_env`](../configuration.md#build) and the per-processor field | the distributions installed in `./.venv` and its interpreter version — see [Python Environment Fingerprint](python-env-fingerprint.md) |

The component *kind* is part of the digest, so a `config` value and a `tool`
value that happen to be equal do not collide. Order is significant and
deterministic (discovery, then analyzers, then tool versions, then the
Python environment).

`rsconstruct product show <path>` prints the components individually, so any
descriptor-key change is attributable to exactly one of them (or to
`input_checksum`).

### Multi-format processors

For processors that produce multiple output formats from the same input (e.g., pandoc producing PDF, HTML, and DOCX), each format is a separate product with a separate cache key. The output format is mixed into the key as the `variant` component, so each format gets its own key naturally.

### Output depends on input name

Most processors produce output that depends only on input content, so the descriptor key hashes input *content*, not input *path*. A rename with identical content is a cache hit by design: the blob is path-free and is restored to the product's current output path (see the `cache_survives_input_rename` test), and two identical files at different paths share one entry.

That is wrong for a processor whose result depends on the path: one that
embeds the input filename in its output (a `// Generated from foo.c` header)
would restore output still naming the old file, and a checker with per-path
rules (ruff's `per-file-ignores`) could reuse another path's verdict. Such a
processor sets `output_depends_on_input_name = true`, a universal field like
`enabled`. ruff, pylint and mypy default it to `true` through their plugin's
`ProcessorDefaults` (their verdicts depend on module names, per-file rules and
the presence of `__init__.py`); a user's explicit value wins. The builder
reads it from the declared config after discovery and marks the instance's
products (`BuildGraph::key_by_input_paths`); `Product::descriptor_key` then
hashes the input paths together with the input checksum. The paths are taken
from the product's current inputs at key time, so inputs an analyzer adds
mid-build are covered too. Turning the field on yields new keys, so the first
build after it rebuilds. Tests: `renamed_input_restores_by_default` and
`output_depends_on_input_name_rebuilds_renamed_input`
(`tests/tests_mod/incremental.rs`).

## Flows

### Lookup

1. Compute the cache key from processor name + config + input contents
2. Look up the object at that key in the object store
3. If not found: cache miss, product must be built
4. If found: read the descriptor, act based on type

### Cache (after successful build)

**Checker:**
1. Store a `{"type": "marker"}` entry at the cache key

**Generator, or mass generator product (single output):**
1. Store the output file content as a content-addressed blob
2. Store a `{"type": "blob", "checksum": "..."}` entry at the cache key

A mass generator runs its tool once, but every predicted file is its own product, so each gets its own blob entry.

**Several outputs, or `output_dirs` (explicit stanzas):**
1. Walk the declared output files and output directories, skipping files declared as outputs of other products
2. Store each file as a content-addressed blob
3. Build the tree entries: `[{"path": "...", "checksum": "...", "mode": ...}, ...]`
4. Store a `{"type": "tree", "entries": [...]}` entry at the cache key

### Restore

**Checker:** Nothing to restore. Cache entry exists = check passed.

**Generator:**
1. Read the cache entry, get the blob checksum
2. Hardlink or copy the blob to the output path

**Tree (several outputs or `output_dirs`):**
1. Read the cache entry, get the tree entries
2. For each `(path, checksum, mode)`: restore the blob to the path, set permissions

### Skip

If the cache entry exists AND all output files are present on disk, no work is needed.

### Inspection is read-only

`status`, `product show`, `build --dry-run` and `--explain` report what the
build would do without doing any of it, and that includes the cache: they
must not populate `.rsconstruct/`. Their lookup (`explain_descriptor`) reads
the local descriptor, or — with `remote_pull` on — downloads and validates
the remote one without writing it locally (`get_descriptor_peeking`), and
checks objects with `object_available`, which asks the remote whether the
object exists rather than fetching it. The build path
(`needs_rebuild_descriptor`, `can_restore_descriptor`,
`restore_from_descriptor`) uses `get_descriptor_pulling` and `ensure_object`,
which persist what they pull because the restore that follows needs it.

The verdict is the same on both paths: a tree descriptor is checked against
the product's *declared* outputs as well as its recorded entries, so an
output the product declares but the last run never produced reads as
"restore" (or "rebuild", if an object is missing) in `status` exactly as it
does in the build.

## Rebuild classification

| Classification | Condition | Action |
|---|---|---|
| **Skip** | Cache key found AND all outputs exist on disk | No work needed |
| **Restore** | Cache key found BUT some outputs are missing | Restore from object store |
| **Build** | No cache entry for this key | Execute the processor |

Because the cache key incorporates input content, a changed input produces a different key. There's no "stale entry" — either the key exists or it doesn't.

### Stale outputs before a rebuild

Cache objects are read-only and a hardlink restore shares their inode, so a restored output cannot be overwritten in place by the tool that regenerates it. Before a product is built or restored, the files of its previous tree are unlinked. That tree cannot be found through the current key (the input changed, so the key did too): the `product_last_tree` table in `db.redb` maps a product's *owner key* — processor instance, primary input and output paths, nothing that changes with content or config — to the descriptor it last built or restored. Outputs left by an rsconstruct older than that table are recognised by the restore's own signature instead: read-only files with a link count above one.

A product that runs under `execution_model = "sandbox"` does not unlink them beforehand: the sandbox removes them inside its overlay as the first step of the run, so a rejected or failed run leaves them in place (see the sandbox design notes, `internal/sandbox.md`).

A product predicted to change that never ran has its stale outputs removed at the end of the run when the build was interrupted. When a product failed, nothing is removed: the products that never ran are the ones the failure blocked, and they keep their outputs. The build is red, and the next build rebuilds them.

## Config-aware caching

Processor configuration is hashed into cache keys. Changing a config value triggers rebuilds even if source files haven't changed.

## Cache restoration methods

| Method | Behavior | Best for |
|---|---|---|
| `hardlink` | Links output to cached blob (same inode, read-only) | Local development (fast, no disk space) |
| `copy` | Copies cached blob to output path (writable) | CI runners, cross-filesystem setups |
| `auto` (default) | Uses `copy` when `CI=true`, `hardlink` otherwise | Most setups |

Hardlinks work because blob objects contain raw file content (not wrapped in a descriptor). Only cache entries (which point to blobs) contain JSON metadata.

## Cache commands

| Command | Description |
|---|---|
| `rsconstruct cache size` | Show cache size and object count |
| `rsconstruct cache list` | List all cache entries as JSON |
| `rsconstruct cache stats` | Show per-processor cache statistics |
| `rsconstruct cache stale` | Show which cache entries are stale vs current |
| `rsconstruct cache remove-stale` | Remove stale entries, then the blobs only they referenced |
| `rsconstruct cache trim` | Remove unreferenced objects |
| `rsconstruct cache clear` | Delete the entire cache |

## Cache growth

The cache only ever grows during a build. Nothing deletes old entries
automatically: there is no size cap, no expiry and no garbage collection
during `build`.

A product's cache key includes its input content, so each input change gives
a new descriptor key and a new set of blobs. The entry for the earlier inputs
stays in place: it is *stale* (no current product computes its key), but its
descriptor still references its blobs. For products whose outputs are small
this does not matter. For products with large outputs that change on almost
every edit, it adds up fast.

**Observed 2026-11-08.** `.rsconstruct/` in this repo had reached 11 GB.
Almost all of it was 46 blobs of about 237 MB each. Every one was a debug
build of the `rsconstruct` binary produced by
`processor.mass_generator.cargo` (profile `dev`): each code change makes a
new binary, the old one stays referenced by its stale descriptor, and so
45 out-of-date binaries had piled up. Across the fleet the same cleanup
freed about 15 GB (book-openbook alone went from 8.4 GB to about 0.3 GB).

**Cleaning up.** Run, in the project:

```bash
rsconstruct cache remove-stale   # drop stale descriptors and the blobs only they referenced
rsconstruct cache trim           # drop any other unreferenced blobs
```

`cache trim` on its own frees nothing here: it removes only blobs that no
descriptor references, and a stale descriptor still references its blobs.
`remove-stale` is the step that releases them. Current entries are kept, so
the next build still restores instead of rebuilding. Use `cache clear` only
for a full reset; the next build starts cold.

**Measuring.** `du -sh .rsconstruct` per repo is accurate for `objects/`, but
`tools/` trees are hardlinked between repos, so adding up per-repo `du`
results overcounts them many times. To measure real disk use across several
repos, run a single `du -cs` over all of them.

**Open issue.** Cleanup is manual, so a cache with large, often-changing
outputs grows by one output-size per change until someone runs
`remove-stale`. Possible lasting fixes: have `build` drop stale entries for
the products it just rebuilt, or add a cache size limit to `BuildConfig`.
Neither exists yet.

## Clean vs Clear

**`rsconstruct clean`** removes build outputs but preserves the cache:

- **Generators**: Output files deleted. Next build restores via hardlink/copy.
- **Checkers**: Nothing to delete. Next build skips.
- **Mass generators**: Every predicted file deleted, like a generator's. Next build restores each from its blob.
- **Explicit**: Declared output files deleted, declared `output_dirs` removed. Next build restores from tree.

**`rsconstruct cache clear`** wipes everything — descriptors and blobs. A cleared cache means "forget everything, rebuild from scratch." The entire `.rsconstruct/` directory is removed, which also takes the auxiliary caches (`mtime.redb`, `deps.redb`, `webcache.redb`) with it. If only blobs were cleared but descriptors survived, the cache would think outputs are available but fail to restore them. Clearing both together avoids this inconsistency.

## Incremental rebuild after partial failure

Each product is cached independently after successful execution. If a build fails partway through, the next run only rebuilds products without valid cache entries.

## Remote caching

See [Remote Caching](../remote-caching.md) for sharing cache between machines and CI.