pub struct JsonParserConfig {
pub format: Option<JsonFormat>,
pub max_rows: Option<usize>,
pub error_policy: JsonErrorPolicy,
}Expand description
Configuration for JSON/JSONL parsing and scanning.
The default detects the grammar from the input and reads every record,
skipping malformed ones. The two format constructors say which grammar to
read with, which is what a caller who knows should do: detection cannot tell
a JSON object document from a JSONL file, since both open with {.
§Examples
The same two records under each grammar:
use std::io::Cursor;
use dataprof_json::{JsonParserConfig, scan_json_from_reader};
let document = r#"[{"id": 1}, {"id": 2}]"#;
let lines = "{\"id\": 1}\n{\"id\": 2}\n";
let from_document = scan_json_from_reader(
Cursor::new(document),
&JsonParserConfig::json_document(),
|_| {},
)?;
let from_lines =
scan_json_from_reader(Cursor::new(lines), &JsonParserConfig::jsonl(), |_| {})?;
assert_eq!(from_document.rows_read, 2);
assert_eq!(from_lines.rows_read, 2);Fields§
§format: Option<JsonFormat>The grammar to read with. None detects it from the first byte, which
resolves [ to JsonFormat::Json and everything else to
JsonFormat::Jsonl — a leading { opens both a JSON object document
and a JSONL file, so callers that know which they have (from the file
extension or an explicit format=) should say so rather than rely on
detection.
max_rows: Option<usize>Maximum rows to process (None = all rows).
error_policy: JsonErrorPolicyHow to react to a malformed record (default: skip).
Implementations§
Source§impl JsonParserConfig
impl JsonParserConfig
Sourcepub fn with_max_rows(self, max_rows: usize) -> Self
pub fn with_max_rows(self, max_rows: usize) -> Self
Set the maximum number of rows to process.
The cap counts profileable records, not lines: blank lines and records skipped as malformed do not consume it.
A cap is only truncation when a record still remained once it was
reached, so a source holding exactly max_rows records reads as
complete. JsonScanSummary::truncated carries that distinction, and
analyze_json_file turns it into a TruncationReason::MaxRows on
the report.
§Examples
use std::io::Cursor;
use dataprof_json::{JsonParserConfig, scan_json_from_reader};
let lines = "{\"id\": 1}\n{\"id\": 2}\n{\"id\": 3}\n";
// Two of three records: a third remained, so the scan was cut short.
let capped = JsonParserConfig::jsonl().with_max_rows(2);
let summary = scan_json_from_reader(Cursor::new(lines), &capped, |_| {})?;
assert_eq!(summary.rows_read, 2);
assert!(summary.truncated);
// A cap the source never reaches past is a complete read, not a
// truncated one.
let exact = JsonParserConfig::jsonl().with_max_rows(3);
let summary = scan_json_from_reader(Cursor::new(lines), &exact, |_| {})?;
assert_eq!(summary.rows_read, 3);
assert!(!summary.truncated);Sourcepub fn with_error_policy(self, policy: JsonErrorPolicy) -> Self
pub fn with_error_policy(self, policy: JsonErrorPolicy) -> Self
Set how malformed records are handled.
Under the default JsonErrorPolicy::Skip a malformed record is
counted in JsonScanSummary::malformed_lines and scanning continues,
so a partial profile never looks like a clean one. Under
JsonErrorPolicy::Strict the first one fails the scan.
§Examples
use std::io::Cursor;
use dataprof_json::{JsonErrorPolicy, JsonParserConfig, scan_json_from_reader};
let lines = "{\"id\": 1}\nnot json\n{\"id\": 2}\n";
let skipping = JsonParserConfig::jsonl();
let summary = scan_json_from_reader(Cursor::new(lines), &skipping, |_| {})?;
assert_eq!(summary.rows_read, 2);
assert_eq!(summary.malformed_lines, 1);
let strict = JsonParserConfig::jsonl().with_error_policy(JsonErrorPolicy::Strict);
assert!(scan_json_from_reader(Cursor::new(lines), &strict, |_| {}).is_err());Sourcepub fn json_document() -> Self
pub fn json_document() -> Self
Read as a standard JSON document: an array of records, or a single object as one record.
Sourcepub fn with_format(self, format: JsonFormat) -> Self
pub fn with_format(self, format: JsonFormat) -> Self
Set the grammar explicitly.