Module nested 

Nested profiles of JSON records (#705): every path to a value, how many records hold it, and the mix of JSON types found there.

A JsonProfiler walks each record (one JSON value per line, as jsonl reads them) and profiles each path as a column: user.id for a key inside an object, tags[] for the items of an array and items[].sku for a key of the objects in one. A record that is not an object is the path value, as in a flat profile.

Path syntax. A key is written as it is unless it is empty, has whitespace at either end, or holds ., [, ], ", \ or a control character; then it is a JSON string in brackets, as JSONPath and jq write it: meta["k8s.io/name"], ["a.b"].c, [""]. path_key writes one.

Bounded. A path has at most ProfileOptions::depth steps (a key or []; default DEFAULT_DEPTH): an object or array at the last step is not walked into, and its compact JSON text is its value, as in a flat profile. At most ProfileOptions::max_paths paths are profiled (default DEFAULT_MAX_PATHS), in the order they are first seen; paths seen after that are named (the first hundred) and noted, and nothing below them is walked. Each path keeps at most ProfileOptions::sample of its values, a fixed-seed reservoir as Profiler keeps rows. Presence, type counts, nulls and the missing-value map count every record. Time is linear in the input; memory is the paths and their samples, however long it is.

Presence and types. For each path the profile counts the records that hold it (ColumnProfile::present) apart from those that do not (absent); an explicit null is present, and counted in nulls. It counts each value’s JSON type (TypeCounts: string, number, boolean, object, array, null), so a field that is sometimes a number and sometimes a string says so instead of being one inferred type; such a path is mixed. Only JSON’s null is null: the null tokens of a flat profile (NA, an empty string) are strings here.

use rich_data::profile::{Profile, ProfileOptions};

let text = r#"{"id": 1, "user": {"name": "ada"}, "tags": ["a", "b"]}
{"id": "2", "user": null}
not json
{"id": 3, "tags": []}
"#;
let profile = Profile::from_json_lines(text.as_bytes(), ProfileOptions::default()).unwrap();
assert!(profile.is_nested());
let names: Vec<&str> = profile.columns().iter().map(|c| c.name.as_str()).collect();
assert_eq!(names, ["id", "user", "user.name", "tags", "tags[]"]);

let id = profile.column("id").unwrap();
assert_eq!(id.types.unwrap().mix(), "num 67% · str 33%");
assert_eq!(id.mixed, Some(true));
let user = profile.column("user").unwrap();
assert_eq!((user.present, user.absent, user.nulls), (Some(2), Some(1), 1));
assert_eq!(profile.column("tags[]").unwrap().values, Some(2));
assert_eq!((profile.malformed(), profile.malformed_lines()), (1, &[3][..]));

Structs

JsonProfiler

Profiles JSON records path by path in bounded memory. See the module docs.

PathSummary

A nested profile’s limits, and what met them.

TypeCounts

How many values at a path were each JSON type. Serialised with only the types seen.

Constants

DEFAULT_DEPTH

The deepest path a nested profile describes by default, in steps.

DEFAULT_MAX_PATHS

The most paths a nested profile describes by default.

MAX_MALFORMED_LINES

The most malformed line numbers a profile keeps.

Functions

path_key

A key as a path writes it: as it is, or as a JSON string in brackets (["k8s.io/name"]) when it is empty, has whitespace at either end, or holds ., [, ], ", \ or a control character. See the module docs.