Module jsonl_proc 

NDJSON / JSON Lines structured processor.

Processes files where each non-empty line is an independent JSON object (Newline-Delimited JSON, also called JSON Lines). Unlike the JsonProcessor, this processor never builds a full in-memory parse tree for the whole file — each line is parsed, walked, serialised, and written out independently, keeping per-line memory overhead constant regardless of input size.

When used via the CLI with a matching profile, the processor is invoked through the streaming path: the file is opened as a reader and processed line-by-line without fs::read loading it into a Vec<u8> first. This makes GB-scale NDJSON log files practical to sanitize.

Options

Key	Values	Default	Description
skip_invalid	"true" / "false"	"false"	Pass malformed lines through unchanged instead of returning an error. Useful for mixed log files that interleave plain-text lines with JSON.
compact	"true" / "false"	"true"	Serialise each output line as compact JSON. Set to "false" only for debugging — pretty-printed NDJSON is non-standard.

Example profile entry

- processor: jsonl
  extensions: [".jsonl", ".ndjson", ".log"]
  options:
    skip_invalid: "true"
  fields:
    - pattern: "*.email"
      category: email
    - pattern: "*.password"
      category: "custom:password"

Structs

JsonLinesProcessor

Structured processor for NDJSON / JSON Lines files.