Module json_crush 

Deterministic JSON crusher — single source of truth for structural JSON compaction (#934, Headroom “Smart Crusher” port, GitLab #935).

Real JSON payloads (API responses, kubectl get -o json, DB dumps, RAG chunks) are dominated by arrays of objects that repeat the same keys and values on every row. This module factors that redundancy out:

• crush_lossless hoists every key that is present in all objects of an array to its dominant value (a _defaults block); each item then keeps only the fields that deviate from the default. A field absent from an item means “equals the default”, so the transform is exactly reconstructible via reconstruct.
• crush_lossy additionally drops near-unique high-entropy columns (timestamps, UUIDs — pure noise for an agent) recorded in _dropped. The exact original is then recovered out-of-band via CCR, never from the text.

Determinism (#498): the output is a pure function of the input Value — no timestamps, counters, randomness, or hash-map order leakage (candidate keys are walked through a BTreeSet, value frequencies through a BTreeMap). The crusher never inflates: callers gate on shorter_only, and a no-op input returns None.

Structs

CrushOpts

Tuning for a crush pass.

CrushResult

Result of a crush pass.

Constants

KEEP_DATA_DIVISOR

Shared “the crush must at least halve the payload” threshold. The lossless crusher keeps every datum, so reshaping only pays when the array is redundant enough that the compact form is at most 1/KEEP_DATA_DIVISOR of the input; heterogeneous/low-redundancy data falls through to each caller’s own outline. Single source for the shell (json_schema, curl) and read (structured_read) paths so the gate can never drift (#936).

Functions

crush_lossless

Lossless crush: returns Some only when something was actually factored.

crush_lossy

Lossy crush: lossless factoring plus high-entropy column dropping.

crush_text_if_beneficial

Parse text as JSON and losslessly crush it, returning the compact form only when it at least halves the input (KEEP_DATA_DIVISOR). The shared gate for callers that start from raw text (structured_read, ctx_read aggressive). None for non-JSON or low-redundancy input — the caller keeps its own path.

crush_text_lossy_if_beneficial

Lossy crush of text: drops near-unique high-entropy columns (timestamps, UUIDs — noise for an agent) whose distinct-value ratio is >= drop_entropy. Returns the CrushResult only when the pass actually dropped a column (!lossless, so a lossless pass would not already cover it) AND the compact form at least halves the input (KEEP_DATA_DIVISOR). Because data is lost, the caller MUST persist the verbatim original out-of-band (CCR) before emitting it — the dropped columns are never reconstructible from the text. None for non-JSON, low-redundancy, or all-lossless input.

crush_value_if_beneficial

Lossless crush of value, returning the compact text only when it at least halves raw_len (KEEP_DATA_DIVISOR). The shared gate for callers that already hold a parsed Value plus its source length (json_schema, curl).

reconstruct

Rebuild a Value from crushed text. Exact for lossless forms; for lossy forms the _dropped columns are simply absent (recover them via CCR).