Module json_prune 

json_prune — Phase 1 opt-in lossy JSON array-item selection (canonical id "json_prune"). Unlike every other transform in this module, this one is NOT lossless: it drops array items to hit a token budget, replacing each with a recoverable {"$tf_ref":...} marker. It is never part of the default lossless pipeline (modes.rs/apply_transforms) — see pipeline::apply_lossy_reduction, which only runs it when policy.lossy.is_some(), as a terminal stage strictly after the normal lossless transform loop.

Full design/rationale: docs/solution-design/lossy-json-compression.md (gitignored, local-only). Summary of the algorithm implemented here:

• Explicit preserve, not inferred force-keep: an array survives untouched iff its path is listed in LossyOptions.preserve_paths. There is no generic “this row is important” inference — that was verified to not exist anywhere in this codebase and to not be invent-able generically.
• Static per-item rank: value-aware structural failure signal (typed field checks, not substring matching) > per-array-field median-absolute-deviation numeric outlier > edge position > original index. Compared lexicographically, no weighted sum, so no cross-scale calibration problem.
• Diversity as a separate allocator step: candidates are grouped by a coarse structural fingerprint and walked round-robin across groups (in each group’s own static-rank order), not folded into the static rank tuple — diversity is set-dependent, the rank tuple isn’t.
• One global, shared budget over every eligible array’s items combined (not an independent ratio per array), spent via a deterministic greedy walk mirroring eval/run_baselines.py::allocate()’s three-tier fit check (proven-safe byte bound → tested-safe heuristic-plus-margin bound → exact re-tokenize fallback) — never summed independent per-item token estimates, which are not additive across tokenizer boundaries.
• Deterministic: no randomness anywhere; the same input always produces the same output.

This module never touches RetrievalStore — per transforms/mod.rs’s convention (“the pipeline owns all bookkeeping”), prune only decides which items to drop and returns their original bytes; pipeline::apply_lossy_reduction does the actual (fail-closed) storing and marker substitution.

Structs

DroppedItem

One dropped item’s original bytes, keyed by its content hash (== the hash the marker substituted in its place points at). pipeline::apply_lossy_reduction persists these fail-closed: an item is only actually removed from the output if RetrievalStore::store returns Ok for it.

LossyOptions

PruneOutcome

PruneReport

Enums

JsonPruneError

Constants

TRANSFORM_ID

TRANSFORM_VERSION

Functions

prune

Runs tiered lossy selection over input. Returns Ok(None) when there’s nothing eligible to prune (no arrays of length >= [MIN_ARRAY_LEN] outside preserved paths, or ratio >= 1.0) — callers should treat that as a clean no-op, not force an empty transform report.

revert_markers

Puts specific dropped items back, keyed by content hash — the fail-closed half of the contract (pipeline::apply_lossy_reduction): an item whose RetrievalStore::store call failed must not be silently lost, so the caller restores it here rather than leaving its $tf_ref marker in place. Recognizes exactly the marker shape [marker_json] builds; any other node (including one that merely looks similar) is left untouched.