# Requirements Generator — Design
A processor that scans Python source files and produces a `requirements.txt`
listing the third-party distributions the project imports. Fills the gap
between the Python analyzer (which discovers local dep edges) and the `pip`
processor (which consumes `requirements.txt`).
## Problem
Users have Python projects with `import` statements. They want the set of
PyPI distributions their code needs, written out to `requirements.txt`.
Today they maintain this file by hand, which drifts from the actual imports.
## Shape
A whole-project `Generator` processor named `requirements`:
- **Inputs:** every `.py` file in the project (same scan as the Python
analyzer — `file_index.scan(&self.config.standard, true)`).
- **Output:** a single `requirements.txt` (path configurable).
- **Discovery:** one `Product` with all `.py` files as inputs, one output
path. Structurally identical to the `tags` processor.
## The classification problem
Every `import X` lands in one of three buckets:
1. **Local** — a module that resolves to a file in the project. Skip.
2. **Stdlib** — a module shipped with Python (`os`, `sys`, `json`, …). Skip.
3. **Third-party** — a PyPI distribution. Emit to `requirements.txt`.
The Python analyzer already resolves bucket 1 via
`PythonDepAnalyzer::resolve_module`. The new processor needs buckets 2 and 3.
### Stdlib detection
Python 3.10+ ships `sys.stdlib_module_names` — a frozenset of every stdlib
top-level module name. We bake this list into a static table inside the
processor (`src/processors/generators/requirements.rs`) rather than probing
`python3` at build time. Reasons:
- The list is stable across 3.10+ with a handful of additions per minor
release.
- No tool dependency at build time — keeps the processor offline and
hermetic.
- The list is ~300 names, a few KB of source.
A refresh script regenerates the table from `python3 -c 'import sys;
print(sorted(sys.stdlib_module_names))'` when we bump Python support. The
list lives alongside the processor, not in a user-facing config.
### Import → distribution mapping
The import name is not always the PyPI distribution name:
| `cv2` | `opencv-python` |
| `yaml` | `PyYAML` |
| `PIL` | `Pillow` |
| `sklearn` | `scikit-learn` |
| `bs4` | `beautifulsoup4` |
We bake a curated table of the common ~40 mismatches into the processor and
default everything else to identity (`import X` → distribution `X`). Users
override via config:
```toml
[processor.requirements.mapping]
cv2 = "opencv-python"
custom_internal = "our-private-dist"
```
User entries win over the built-in table. This is lossy by design — we
accept that unusual packages need a config entry — in exchange for:
- No dependency on an installed Python environment.
- `requirements.txt` generation works on a clean checkout (no
chicken-and-egg with `pip install`).
- Deterministic output regardless of the caller's environment.
The alternative — probing `importlib.metadata.packages_distributions()` —
is more accurate but requires packages to already be installed. Rejected
for now; can be added later as an opt-in `resolve = "probe"` mode if users
hit the mapping ceiling.
## Configuration
```toml
[processor.requirements]
output = "requirements.txt" # Output file path
exclude = [] # Import names to never emit (e.g. internal vendored modules)
sorted = true # Sort output alphabetically (vs. discovery order)
header = true # Emit a "# Generated by rsconstruct" header line
[processor.requirements.mapping]
cv2 = "opencv-python" # User-provided import → distribution overrides
```
| `output` | string | `"requirements.txt"` | Output file path |
| `exclude` | string[] | `[]` | Import names to never emit |
| `sorted` | bool | `true` | Sort entries alphabetically |
| `header` | bool | `true` | Include a comment header line |
| `mapping` | map | `{}` | Per-project import→distribution overrides |
Pinning (`pkg==1.2.3`) is deferred. The first iteration emits bare names.
Adding pinning later means probing `pip show` or parsing a lockfile —
separate concern.
## Code organization
### Shared import scanner
Factor the regex scanning out of `src/analyzers/python.rs` into a module
function shared by the analyzer and the generator:
```rust
// src/analyzers/python.rs
pub(crate) fn scan_python_imports(path: &Path) -> Result<Vec<String>> { ... }
```
Returns the raw top-level module names found in `import X` and `from X
import ...` lines. The analyzer then runs this through `resolve_module` to
keep local ones; the generator runs it through the stdlib table and
mapping to produce the final list.
This fixes architecture-observations #6 (analyzers can't hand data to
processors) at the scope of this one feature: instead of building a
cross-processor channel, we share a pure function.
### Files
- `src/processors/generators/requirements.rs` — the processor, plus its two
data tables: the stdlib names table (static `&[&str]`) with an
`is_stdlib(module: &str) -> bool` helper, and the curated
import→distribution mapping with a `resolve_distribution(import: &str) ->
&str` helper that falls through to identity. Every file under
`src/processors/` must be a real processor, so the tables live in the
processor's own file.
- `src/config/processor_configs.rs` — add `RequirementsConfig`.
- `src/processors/mod.rs` — add `pub const REQUIREMENTS = "requirements"`
to `names` module.
- `docs/src/processors/requirements.md` — user-facing processor doc.
### Processor structure
Mirrors `tags` (whole-project generator with one output):
```rust
pub struct RequirementsProcessor {
base: ProcessorBase,
config: RequirementsConfig,
}
impl Processor for RequirementsProcessor {
fn discover(&self, graph, file_index, instance_name) -> Result<()> {
// Scan for .py files; if none, no product.
// Add one product: inputs=all .py files, outputs=[output_path].
}
fn supports_batch(&self) -> bool { false }
fn execute(&self, _ctx, product) -> Result<()> {
// 1. Scan each input .py for imports.
// 2. For each top-level module name:
// - Skip if local (resolves to a project file).
// - Skip if stdlib.
// - Skip if in user's `exclude`.
// - Map import → distribution name.
// 3. Dedupe, sort if configured, write to output.
}
}
```
## Cache behavior
Falls naturally out of the descriptor-based cache:
- Inputs: every `.py` file + config hash.
- Output: `requirements.txt`.
- Adding/removing an import changes file contents, triggers rebuild.
- Changing config (new mapping entry, new exclude) changes config hash,
triggers rebuild.
- Code changes inside a function that don't affect imports still trigger a
rebuild, since we can't cheaply know which lines matter. Acceptable —
the regeneration is fast.
## Auto-detection
`auto_detect` returns `true` when the file index contains any `.py` files.
Same criterion as the Python analyzer.
## Out of scope (first cut)
- Version pinning.
- Multiple output files (`requirements-dev.txt`, `requirements-test.txt`).
- Optional dependencies / extras (`pkg[extra]`).
- Reading existing `requirements.txt` to preserve comments or pins.
- `pyproject.toml` or `setup.py` output — `requirements.txt` only.
Each is a clean follow-up if users ask.