rsconstruct 0.9.172

Rust based fast build system
# Completed Suggestions

Items from `suggestions.md` that have been implemented.

## Completed Features

- **Remote caching** — See [Remote Caching](../remote-caching.md). Share build artifacts across machines via S3, HTTP, or filesystem.
- **Lua plugin system** — built, then removed on 2026-10-17: no repository ever declared a `[processor.lua.*]` stanza, and the host could not take part in cache versioning or field validation. Custom commands go through the config-driven processors (`script`, `generic`) instead.
- **Tool version locking** — `rsconstruct tool lock` locks and verifies external tool versions. Tool versions are included in cache keys.
- **JSON output mode** — `--json` flag for machine-readable JSON Lines output (build_start, product_start, product_complete, build_summary events).
- **Builtin C/C++ include scanner** — Default `include_scanner = "builtin"` uses regex-based scanning. Falls back to `include_scanner = "compiler"` (gcc -MM).
- **`--processor` flag** — `rsconstruct build -p processor.generator.tera,processor.checker.ruff` and `rsconstruct watch -p processor.generator.tera` filter which processors run.
- **Colored diff on config changes** — When processor config changes trigger rebuilds, rsconstruct shows what changed with colored diff output.
- **Batch processing** — ruff, pylint, shellcheck, zspell, mypy, and rumdl all support batch execution via `execute_batch()`.
- **Progress bar** — Uses `indicatif` crate. Progress bar sized to actual work (excludes instant skips), hidden in verbose/JSON mode.
- **Emit `ProductStart` JSON events** — Emitted before each product starts executing, pairs with `ProductComplete` for per-product timing.
- **mypy processor** — Python type checking with mypy. Batch-capable. Auto-detects `mypy.ini` as extra input.
- **Explain commands** — `--explain` flag shows skip/restore/rebuild reasons for each product during build.
- **`-p` pulls in the producers of its inputs** — `-p B` runs B and the processors that generate B's inputs, transitively, as `--target` does for products; every processor is discovered and the selection is made on the full graph (`select_processors`). `-x` excludes a processor and everything downstream of it, and an excluded processor is never pulled back in.
- **Unchanged-output pruning** — When a product rebuilds to identical bytes, the products reading it are skipped with their outputs untouched: no rerun, no restore. Each product is decided at dispatch from the inputs it will read (R1), and its stale outputs are removed only right before it actually restores or rebuilds — no longer for every predicted change before the run. Products predicted to change that never run because of Ctrl+C have their stale outputs removed at the end of the run; after a failure the products it blocked keep theirs (the build is red, and the next build rebuilds them). Processors whose one tool run writes many products' outputs (mass generators) replace them themselves (`Processor::replaces_own_outputs`).
- **Ready-queue scheduler** — Products run the moment their own dependencies finish, from a pool of exactly `-j` workers, instead of level by level. Batch chunks run in parallel and count against `-j`; `max_jobs` is enforced by the queue instead of semaphores. See R6/R7 in [Architecture Observations](architecture-observations.md).
- **Processor sections namespaced by type** — a processor is named by its full path, `processor.<type>.<name>` (`[processor.checker.ruff]`, `[processor.checker.pylint.core]`), and that name is also the `-p` argument, the cache key and the default output directory. See [Processor names](../configuration.md#processor-names).

## Completed Code Consolidation

- **Collapsed `checker_config!` macro variants** — Merged `@basic`, `@with_auto_inputs`, and `@with_linter` into two internal variants (`@no_linter` and `@with_linter`).
- **Added `batch` field to all manually-defined processor configs** — All processor configs now support `batch = false` to disable batching per-project.
- **Replaced trivial checker files with `simple_checker!` macro** — 25 trivial checkers reduced from ~35 lines each to 3-5 lines (~800 lines eliminated).
- **Unified `lint_files`/`check_files` naming** — All checkers now use `check_files` consistently.
- **Moved `should_process` guard into macro** — Added `guard: scan_root` built-in to `impl_checker!`, removed boilerplate `should_process()` from 7 processors.
- **Simplified `KnownFields`** — Scan config fields auto-appended by validation layer via `SCAN_CONFIG_FIELDS` constant; `KnownFields` impls only list their own fields.
- **Extracted `WordManager` for spellcheck/aspell** — Shared word-file management (loading, collecting, flushing, execute/batch patterns) in `word_manager.rs`.
- **One checker discovery helper** — `discover_checker_products` takes `dep_inputs` and `dep_auto` explicitly; the `checker_discover` variant that differed only in merging `dep_auto` is gone, and with it the hazard of picking the wrong one.
- **Remote pull wired in** — `remote_pull` and the `try_fetch_*` helpers are called from the descriptor and restore paths on a local miss; no unused scaffold remains.
- **`processor_type` is live** — it is half of every processor's name (`ProcessorPlugin::pname`), not a dead field.
- **Processors own their tools** — each processor file submits the `ToolInfo` of the tools only it runs; the central `TOOLS` table in `src/tools.rs` holds only what no single processor owns, enforced by `central_registry_holds_only_unowned_tools`.
- **Batch timings carry `start_offset`** — both batch paths in `src/executor/execution.rs` record it, so `--trace` places batched products correctly.

## Completed New Processors

- **mypy** — Python type checking using `mypy`. Batch-capable. Config: `checker`, `args`, `dep_inputs`, `scan`.
- **yamllint** — Lint YAML files using `yamllint`. `src/processor/checker/yamllint.rs`.
- **jsonlint** — Validate JSON files for syntax errors. `src/processor/checker/jsonlint.rs`.
- **taplo (toml-lint)** — Validate TOML files using `taplo`. `src/processor/checker/taplo.rs`.
- **markdownlint** — Lint Markdown files for structural issues. Uses `mdl` or `markdownlint-cli`.
- **pandoc** — Convert Markdown to other formats (PDF, HTML, EPUB). Generator processor.
- **jinja2** — Render Jinja2 templates (`.j2`) via Python jinja2 library. `src/processor/generator/jinja2.rs`.
- **black** — Python formatting verification using `black --check`. `src/processor/checker/black.rs`.
- **rust_single_file** — Compile single-file Rust programs to executables. `src/processor/generator/rust_single_file.rs`.
- **sass** — Compile SCSS/SASS files to CSS. `src/processor/generator/sass.rs`.
- **protobuf** — Compile `.proto` files to generated code using `protoc`. `src/processor/generator/protobuf.rs`.
- **pytest** — Run Python test files with pytest. `src/processor/checker/pytest.rs`.
- **doctest** — Run Python doctests via `python -m doctest`. `src/processor/checker/doctest.rs`.

## Completed Test Coverage

- **Ruff/pylint processor tests** — `tests/processor/ruff.rs` and `tests/processor/pylint.rs` with integration tests.
- **Make processor tests** — `tests/processor/make.rs` with Makefile discovery and execution tests.
- **All generator processor tests** — Integration tests for all 14 previously untested generators: a2x, drawio, gem, libreoffice, markdown, marp, mermaid, npm, pandoc, pdflatex, pdfunite, pip, sphinx.
- **All checker processor tests** — Integration tests for all 5 previously untested checkers: ascii, aspell, markdownlint, mdbook, mdl.

## Completed Caching & Performance

- **Lazy file hashing (mtime-based)** — `mtime_check` config (default `true`), `fast_checksum()` with MTIME_TABLE. Stores `(path, mtime, checksum)` tuples. Disable with `--no-mtime`.
- **Compressed cache objects** — Optional zstd compression for `.rsconstruct/objects/`. Config: `compression = true` in `[cache]`. Incompatible with hardlink restore (must use `restore_method = "copy"`). Checksums computed on original content for stable cache keys.

## Completed Developer Experience

- **`--quiet` flag** — `-q`/`--quiet` suppresses all output except errors. Useful for CI scripts that only care about exit code.
- **Flaky product detection / retry** — `--retry=N` retries failed products up to N times. Reports FLAKY (passed on retry) vs FAILED status in build summary.
- **Actionable error messages** — `rsconstruct tool check` shows install hints for missing tools (e.g., "install with: pip install ruff").
- **Build profiling / tracing** — `--trace=file.json` generates Chrome trace format output viewable in `chrome://tracing` or Perfetto UI.
- **`rsconstruct build <target>`** — Build specific targets by name or pattern via `--target` glob patterns and `-d/--dir` flags.
- **`rsconstruct why <file>` / Explain rebuilds** — `--explain` flag shows why each product is skipped, restored, or rebuilt.
- **`rsconstruct doctor`** — Diagnose build environment: checks config, tools, and versions. Full implementation in `src/builder/doctor.rs`.
- **`rsconstruct sloc`** — Source lines of code statistics with COCOMO effort/cost estimation. `src/builder/sloc.rs`.

## Completed Quick Wins

- **Batch processing for more processors** — All checker processors that support multiple file arguments now use batching.
- **Progress bar for long builds** — Implemented with `indicatif`, shows `[elapsed] [bar] pos/len message`.
- **`--processor` flag for build and watch** — Filter processors with `-p` flag.
- **Emit `ProductStart` JSON events** — Wired up and emitted before execution.
- **Colored diff on config changes** — Shows colored JSON diff when processor config changes.

## Completed Features (v0.3.7)

- **`RSCONSTRUCT_THREADS` env var** — Set parallelism via environment variable instead of `-j`. Priority: CLI `-j` > `RSCONSTRUCT_THREADS` > config `parallel`.
- **Global `output_dir` in `[build]`** — Global output directory prefix (default: `"out"`). Processor defaults like `out/processor.generator.marp` are remapped when the global is changed (e.g., `output_dir = "build"` makes marp output to `build/marp`). Individual processors can still override their `output_dir` explicitly.
- **Named processor instance output directories** — When multiple instances of the same processor are declared (e.g., `[processor.generator.marp.slides]` and `[processor.generator.marp.docs]`), each instance defaults to `out/{instance_name}` (e.g., `out/marp.slides`, `out/marp.docs`) instead of sharing the same output directory.
- **Named processor instance names in error reporting** — When multiple instances of the same processor exist, error messages, build progress, and statistics use the full instance name (e.g., `[pylint.core]`, `[pylint.tests]`). Single instances continue to use just the processor type name.
- **`processor config` without config file** — `rsconstruct processor config <name>` now works without an `rsconstruct.toml`, showing the default configuration (same as `defconfig`).
- **`tag collect` command** — `rsconstruct tag collect` scans the tags database for tags that are not in the tag collection (`tags_dir`) and adds them to the appropriate `.txt` files. Key:value tags go to `{key}.txt`, bare tags go to `tags.txt`.
- **`rsconstruct status` shows 0-file processors** — Processors declared in the config that match no files are now shown in `status` output and the `--breakdown` summary, making it easy to spot misconfigured or unnecessary processors.
- **`smart remove-no-file-processors`** — New command `rsconstruct smart remove-no-file-processors` removes `[processor.*]` sections from `rsconstruct.toml` for processors that don't match any files. Handles both single and named instances.
- **`cc_single_file` output_dir from config** — The `cc_single_file` processor now reads its output directory from the config `output_dir` field instead of hardcoding `out/processor.generator.cc_single_file`. This fixes named instances (e.g., `cc_single_file.gcc` and `cc_single_file.clang`) which previously collided on the same output directory.
- **`clean unknown` respects .gitignore** — `rsconstruct clean unknown` now skips gitignored files. Previously it disabled .gitignore handling, causing intentionally ignored files (IDE configs, virtualenvs, `*.pyc`, etc.) to be flagged as unknown. RSConstruct outputs are still correctly identified via the build graph, so nothing is missed. Use `--no-gitignore` to include gitignored files.
- **Cross-processor dependencies (fixed-point discovery)** — Generator outputs are now visible to downstream processors on the first build. Discovery runs in a fixed-point loop: after each pass, declared outputs are injected as virtual files into the FileIndex, and discovery re-runs until no new products are found. This means a generator that creates `.md` files can feed pandoc/tags/spell-checkers in a single build, without needing a second build.

## Completed Architecture Refactors

- **Config provenance tracking** — Every config field now carries `FieldProvenance` (UserToml with line number, ProcessorDefault, ScanDefault, OutputDirDefault, SerdeDefault). `rsconstruct config show` annotates every field with its source. Uses `toml_edit::Document` for span capture.
- **`BuildContext` replacing process globals** — All mutable process globals moved into `BuildContext`: the three processor globals (`INTERRUPTED`, `RUNTIME`, `INTERRUPT_SENDER`) and the three checksum globals (`CACHE`, `MTIME_DB`, `MTIME_ENABLED`). Threaded through the `Processor` trait, executor, analyzers, remote cache, checksum functions, and deps cache. Signal handler uses `Arc<BuildContext>`.
- **`BuildPolicy` trait** — Extracted from the executor. `classify_products` delegates per-product skip/restore/rebuild decisions to a `&dyn BuildPolicy`. `IncrementalPolicy` implements the current logic. Future policies (dry-run, always-rebuild, time-windowed) are a single trait impl.
- **`ObjectStore` decomposition** — `mod.rs` split from 664 → 223 lines into focused submodules: `blobs.rs` (content-addressed storage), `descriptors.rs` (cache descriptor CRUD), `restore.rs` (restore/needs_rebuild/can_restore/explain).

## Completed Features (latest)

- **`rsconstruct status --json`** — JSON output with per-processor counts (`up_to_date`, `restorable`, `stale`, `new`, `total`, `builtin`) and totals. Activated by `--json` flag.
- **Selective processor cleaning** — `rsconstruct clean outputs -p processor.checker.ruff,processor.checker.pylint` cleans only those processors' outputs. Without `-p`, cleans everything.
- **Prettier processor** — Checker using `prettier --check`. Batch-capable. Scans `.js/.jsx/.ts/.tsx/.mjs/.cjs/.css/.scss/.less/.html/.json/.md/.yaml/.yml`. `src/processor/checker/prettier.rs`.
- **Bare `clean` requires subcommand** — `rsconstruct clean` now errors with usage hint instead of silently defaulting to `clean outputs`.
- **Nondeterministic test race fix** — Fixed TOCTOU race in `store_descriptor` where parallel writers could get `Permission denied`. Now retries after forcing writable on first failure.
- **Suppress status line for non-build commands** — The `Exited with SUCCESS/ERROR` footer only shows for `build`, `watch`, and `clean`.
- **Configurable graph validation** — Four checks run after `resolve_dependencies()`: (1) reject empty inputs (default on), (2) validate dep references (default on), (3) detect duplicate inputs within same processor (default off), (4) early cycle detection (default off). Config: `[graph]` section fields `validate_empty_inputs`, `validate_dep_references`, `validate_duplicate_inputs`, `validate_early_cycles`.
- **Checksum globals moved to BuildContext** — `CACHE`, `MTIME_DB`, `MTIME_ENABLED` moved from `src/checksum.rs` statics into `BuildContext`. `combined_input_checksum`, `checksum_fast`, `file_checksum` all take `&BuildContext`. Completes the isolated-build-context story.
- **`rsconstruct fix` command** — Runs fixers (auto-format, auto-fix) on source files. Checkers declare fix capability via `fix_subcommand`/`fix_prepend_args` on `SimpleCheckerParams`. `processor list` shows a `Fix` column. Supports `-p` filtering, batch execution, and `--json`. Fix-capable processors: ruff, black, prettier, eslint, stylelint, standard, taplo, rumdl, markdownlint. *(Partly reverted 2026-08-04: fix capability was unused, so all nine `can_fix` flags were set to `false`; today only `script` instances with a `fix_command` are fix-capable, and processors are named as positional arguments to `rsconstruct fix run` rather than with `-p`. Removed entirely on 2026-10-17: the one fleet use was a `script` instance, and nothing else ever declared a fixer.)*
- **`processor search`** — `rsconstruct processor search <query>` searches by name, description, and keywords. All 91 processors have keywords covering language, tool category, file extensions, and ecosystem terms. Supports `--json` output.