Skip to main content

Module json_dict

Module json_dict 

Source
Expand description

json_value_dict transform (canonical id "json_value_dict", v1.0.0).

Content-aware, losslessly reversible value deduplication for generic JSON data. Where json_field_fold factors out repeated keys, this factors out repeated values: the same large object/array/string appearing many times (a constant nested record, a repeated timestamp, an identical config blob) is stored once in a dictionary and every occurrence is replaced by a compact reference.

{"a":{"x":1,"y":2,"z":3},"b":{"x":1,"y":2,"z":3}}
  -> {"__tf_dict__":[{"x":1,"y":2,"z":3}],
      "__tf_data__":{"a":{"__tf_ref__":0},"b":{"__tf_ref__":0}}}

It runs after json_field_fold in the pipeline, so it also collapses the repeated nested values that folding surfaces across rows. Only values whose serialized form is large enough that a reference is cheaper are dictionaried (see MIN_VALUE_BYTES), and the pipeline’s exact-token gate drops the whole transform if it fails to net a saving — so it can never make a payload worse. Reversibility is guaranteed by round_trips (the pipeline safety gate): a fold that wouldn’t restore exactly is rolled back.

Enums§

JsonDictError

Constants§

TRANSFORM_ID
Canonical transform id, as registered with the pipeline.
TRANSFORM_VERSION
Semantic version of this transform’s output behavior.

Functions§

dict_json
Replaces repeated large values in input with dictionary references. No-op-safe: empty input returns empty; input with nothing worth dictionarying returns unchanged.
round_trips
True iff expanding after reproduces before exactly (as JSON values) — the pipeline’s safety gate for this transform.
undict_json
Inverse of dict_json: expands every {__tf_ref__: i} back to dict[i].