pub fn scan_json_from_reader<R, F>(
reader: R,
config: &JsonParserConfig,
on_object: F,
) -> Result<JsonScanSummary, DataProfilerError>Expand description
Scan JSON or JSONL input and invoke on_object for each object record.
- JSONL: one record per physical line. A record may not span lines and a line may not hold more than one value, so a concatenation whose delimiter was lost is reported rather than read as several clean records. Blank lines are separators: neither records nor errors.
- JSON: one standard document. An array of objects streams element by element without buffering the whole input; a single object is read whole and is one record, which is why it may be pretty-printed across lines.
§Record policy
Only JSON objects are profileable records — they are the only JSON value
with named fields to turn into columns. A record that is valid JSON but not
an object (a scalar, an array, null) is never silently discarded: under
JsonErrorPolicy::Skip it is counted in
JsonScanSummary::malformed_lines and scanning continues with the next
record, and under JsonErrorPolicy::Strict the first one fails the scan.
§Examples
Reading JSONL, keeping one field per record:
use std::io::Cursor;
use dataprof_json::{JsonParserConfig, scan_json_from_reader};
let lines = "{\"city\": \"Rome\", \"pop\": 2800000}\n\
{\"city\": \"Milan\", \"pop\": 1400000}\n";
let mut cities = Vec::new();
let summary = scan_json_from_reader(
Cursor::new(lines),
&JsonParserConfig::jsonl(),
|object| {
if let Some(city) = object.get("city").and_then(|value| value.as_str()) {
cities.push(city.to_string());
}
},
)?;
assert_eq!(cities, ["Rome", "Milan"]);
assert_eq!(summary.rows_read, 2);
assert!(!summary.truncated);The same records as a JSON array. The array streams element by element, and a record may be pretty-printed across lines because whitespace carries no meaning under this grammar:
use std::io::Cursor;
use dataprof_json::{JsonParserConfig, scan_json_from_reader};
let document = r#"[
{"city": "Rome", "pop": 2800000},
{"city": "Milan", "pop": 1400000}
]"#;
let mut populations = Vec::new();
let summary = scan_json_from_reader(
Cursor::new(document),
&JsonParserConfig::json_document(),
|object| {
// A field that is absent, or present but not a number, stays
// `None`. Reading it as 0 instead would turn a missing value into
// a real measurement, which is the one thing a profile must never
// do.
populations.push(object.get("pop").and_then(|value| value.as_u64()));
},
)?;
assert_eq!(populations, [Some(2_800_000), Some(1_400_000)]);
assert_eq!(summary.rows_read, 2);
assert_eq!(summary.malformed_lines, 0);