Skip to main content

scan_json_from_reader

Function scan_json_from_reader 

Source
pub fn scan_json_from_reader<R, F>(
    reader: R,
    config: &JsonParserConfig,
    on_object: F,
) -> Result<JsonScanSummary, DataProfilerError>
where R: BufRead, F: FnMut(&JsonObject),
Expand description

Scan JSON or JSONL input and invoke on_object for each object record.

  • JSONL: one record per physical line. A record may not span lines and a line may not hold more than one value, so a concatenation whose delimiter was lost is reported rather than read as several clean records. Blank lines are separators: neither records nor errors.
  • JSON: one standard document. An array of objects streams element by element without buffering the whole input; a single object is read whole and is one record, which is why it may be pretty-printed across lines.

§Record policy

Only JSON objects are profileable records — they are the only JSON value with named fields to turn into columns. A record that is valid JSON but not an object (a scalar, an array, null) is never silently discarded: under JsonErrorPolicy::Skip it is counted in JsonScanSummary::malformed_lines and scanning continues with the next record, and under JsonErrorPolicy::Strict the first one fails the scan.

§Examples

Reading JSONL, keeping one field per record:

use std::io::Cursor;

use dataprof_json::{JsonParserConfig, scan_json_from_reader};

let lines = "{\"city\": \"Rome\", \"pop\": 2800000}\n\
             {\"city\": \"Milan\", \"pop\": 1400000}\n";

let mut cities = Vec::new();
let summary = scan_json_from_reader(
    Cursor::new(lines),
    &JsonParserConfig::jsonl(),
    |object| {
        if let Some(city) = object.get("city").and_then(|value| value.as_str()) {
            cities.push(city.to_string());
        }
    },
)?;

assert_eq!(cities, ["Rome", "Milan"]);
assert_eq!(summary.rows_read, 2);
assert!(!summary.truncated);

The same records as a JSON array. The array streams element by element, and a record may be pretty-printed across lines because whitespace carries no meaning under this grammar:

use std::io::Cursor;

use dataprof_json::{JsonParserConfig, scan_json_from_reader};

let document = r#"[
    {"city": "Rome", "pop": 2800000},
    {"city": "Milan", "pop": 1400000}
]"#;

let mut populations = Vec::new();
let summary = scan_json_from_reader(
    Cursor::new(document),
    &JsonParserConfig::json_document(),
    |object| {
        // A field that is absent, or present but not a number, stays
        // `None`. Reading it as 0 instead would turn a missing value into
        // a real measurement, which is the one thing a profile must never
        // do.
        populations.push(object.get("pop").and_then(|value| value.as_u64()));
    },
)?;

assert_eq!(populations, [Some(2_800_000), Some(1_400_000)]);
assert_eq!(summary.rows_read, 2);
assert_eq!(summary.malformed_lines, 0);