Module json_sample 

Statistical JSON array row-sampling (#1147, SmartCrusher-equivalent).

While json_crush factors constant/dominant columns out of homogeneous arrays (keeping ALL rows), this module selects a representative subset of rows from large arrays — the same strategy Headroom’s SmartCrusher uses.

The two modules are complementary: json_sample reduces 1000 rows to ~15-30, then json_crush factors the remaining rows’ shared columns. Wired in series by the shell engine’s verbatim-data ladder.

Algorithms:

• Field variance scoring — ranks fields by information content (distinct value ratio + type heterogeneity) to identify which columns carry signal.
• Kneedle-inspired budget — picks the subset size where marginal coverage gain diminishes (bigram coverage over high-signal fields).
• Stratified retention — 30% from head (schema + context), 15% from tail (recency), 55% by importance (anomalies, errors, boundary values).
• Anomaly preservation — rows containing error indicators or statistical outliers on numeric fields are ALWAYS kept regardless of budget.

Determinism (#498): output is a pure function of (array, config). Row selection uses sorted indices and stable tie-breaking on position; no randomness, no hash-map iteration order leakage.

Structs

SampleOpts

Configuration for the sampler.

SampleResult

Result of a sampling pass.

Functions

sample_array

Sample a JSON array, returning the representative subset with a summary. Returns None if the array is too small or not an array of objects.

sample_text_if_beneficial

Parse text as JSON and sample if it’s a large array of objects.