module-harness 0.2.0

Parse, validate, and generate module-level harness context (spec, contracts, evals) for any programming language
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

# module-harness Specification

## Purpose

Provide deterministic tooling for managing module-level harness context — structured documentation that bridges the gap between source code and AI agent understanding.

## Architecture

```
module-harness (binary — deterministic)
├── parse <file>          → JSON of existing harness sections
├── inventory [dir]       → list all evals across modules
├── diff <file>           → show harness summary / drift
├── coverage [dir]        → match evals to test functions (planned)
└── format <file> --lang  → write harness from JSON input (planned)

module-harness (skill — non-deterministic, LLM-powered)
├── /module-harness <file>        → generate/update harness for one module
├── /module-harness batch <dir>   → all modules in directory
└── /module-harness audit <dir>   → report drift and coverage
```

**Binary** handles structure: parsing, validation, inventory, metrics.
**Skill** handles understanding: reading code, writing specs, inferring contracts.

## Harness Sections

### Spec

Behavioral specification. One entry per public function, type, or trait. Describes *what* the module does, not *how*.

- Must cover every public symbol
- Should be verifiable (testable claims, not aspirations)
- Ordered by importance / call frequency

### Agentic Contracts

Invariants and guarantees that callers can rely on without reading implementation.

- Error handling behavior (panics, returns, logs)
- Atomicity and concurrency guarantees
- Idempotency claims
- Side effect boundaries (what the module touches)

### Evals

Named testable scenarios. Each eval has:
- **Name**: snake_case identifier (could be a test function name)
- **Description**: scenario → expected outcome

Evals serve dual purpose:
1. **Documentation** — what should be tested
2. **Metrics** — quantifiable coverage for comparing harness contexts

## Eval Metrics (Planned)

Evals create a quantifiable signal for comparing different harness contexts via backtesting:

- **Coverage ratio**: evals with matching tests / total evals
- **Drift score**: public symbols not mentioned in spec / total public symbols
- **Completeness**: modules with harness / total modules

These metrics enable A/B comparison: given two harness context versions, which one leads to better agent outcomes when working on the codebase?

## Future: Backtesting

The backtesting workflow (not yet implemented):

1. Select a git revision as baseline
2. Present the agent with a target spec (feature to implement, bug to fix)
3. Run with harness context A, measure outcome
4. Run with harness context B, measure outcome
5. Compare: which harness led to fewer errors, less drift, faster completion?

This creates a feedback loop: harness quality improves agent performance, which is measurable.