spate-json

JSON deserialization for the Spate framework. Decodes JSON payloads from a source into your own serde types or dynamically-typed values.

Applications should depend on the spate facade crate with the json feature rather than on this crate directly.

Framings

One payload maps to 0..N records, chosen by framing:

single — the whole payload is one JSON document → one record (the Kafka-message default). Empty/whitespace-only payloads are tombstones.
ndjson — newline-delimited JSON (JSON Lines): one value per \n-separated line → one record per line, with per-line error isolation.
array — a top-level JSON array → one record per element (decoded in one pass; a malformed array is handled atomically).

Those frame a payload already in memory (a Kafka message). For a streaming source that never holds a whole object in RAM (spate-s3, an HTTP body), the crate also owns the byte-stream framer: NdjsonFramer is a chunk-fed, bounded RecordFramer the source runs to cut the stream into records — wire it in with S3Source::with_framer(|| Box::new(NdjsonFramer::new(max))), then the deserializer decodes each line in single mode.

Configuration

deserializer:
  json:
    framing: single              # single | ndjson | array
    on_error: skip               # skip | fail
    reject_duplicate_keys: false

let deserializer = JsonDeserializerBuilder::from_component(deser_section)?
    .with_metrics(ctx.pipeline, "main")
    .build_serde::<Order>();

Errors and metrics

A document that does not parse (or match the target type) is a record-level error. on_error: skip drops it, counts it in spate_json_deser_records_dropped_total{reason}, and continues; on_error: fail stops the batch, which replays under at-least-once. The framework's generic spate_deser_* stage metrics wrap every decoder in addition.

Fidelity features

reject_duplicate_keys (config) turns silent last-value-wins on duplicate object keys into an error. Opt-in Cargo features pass straight through to serde_json: float-roundtrip, arbitrary-precision (crate-wide; interacts with flatten/untagged/RawValue), and raw-value.

Backends