Expand description
Product map: which API for which job (ingest, views, project, enums, bridges).
§API guide — when to use what
jshift exposes several surfaces that look similar (JsonView, path gets,
project, multi-page streams, optional serde/axum bridges). Picking the
wrong one for the job is the main way to get surprising benches (e.g.
serde beating a JsonView card on multi-field ingest).
This guide is the product map. Prefer it over inventing a parallel style.
§One-line product
Schema-guided JSON on raw bytes: sparse path read, open in-place mutate,
project/stream — without a serde_json::Value tree as the spine.
§When to use which API
| Job | Prefer | Avoid for this job |
|---|---|---|
Hot path: 1–few fields (id, seq, flag) | JsonDoc::get / each_get / find_value | Full [JsonView] card, Value |
Ingest / stage tables (cols + optional raw_json) | each_with + find_value (static PathSegments); raw = element span | JsonView multi-string card as the hot loop |
| Domain object (handler, business logic, open mutate) | #[derive(JsonView)] + read_from / write_into | Hand-rolled find_value soup for 20 fields |
| Thin JSON out (drop variants/images) | project / project_jmespath / ProjectPlan | Deserialize full tree then re-serialize |
| Archive bronze (exact wire bytes) | Write the buffer; hold TypedDoc if you need paths later | Parsing into Arrow “for storage” if you only need raw |
| Paginated / multi-file / JSONL | for_each_page_items, for_each_jsonl_multi, JsonlDecoder | Concatenate pages into one mega-buffer |
| HTTP body → view (axum) | feature axum: JshiftJson / JshiftDoc | Json<Value> then convert |
Must satisfy Deserialize | feature serde: AsJshiftView — see serde bridge guide | Implementing Deserialize on every jshift type by default |
| Externally tagged enums | #[derive(JsonView)] default, or read_external_* | Value match on tags |
| Internally / adjacent / untagged enums | #[json(tag = …)] / content / untagged (0.7+) — see crate::enums | Assuming only external exists |
§Ingest pattern (Arrow / Parquet hand-off)
jshift does not build Arrow rows. Your loop does:
use jshift::{find_value, FromJsonSlice, JsonDoc, PathSegment, TypedDocRef};
const P_ID: &[PathSegment] = &[PathSegment::Key("id")];
const P_TITLE: &[PathSegment] = &[PathSegment::Key("title")];
let doc = TypedDocRef::from_slice(page_json);
doc.each_with("products", |elem| {
let id = u64::from_json_slice(find_value(elem, P_ID)?)?;
let title = String::from_json_slice(find_value(elem, P_TITLE)?)?;
let raw_json = std::str::from_utf8(elem)?; // full product span
// id_col.append_value(id);
// title_col.append_value(&title);
// raw_col.append_value(raw_json);
let _ = (id, title, raw_json);
Ok(())
})?;
// finish builders → RecordBatch → Parquet (arrow/parquet crates, not jshift)| Layer | Owner |
|---|---|
| Nested JSON on the wire | Network / files |
| Sparse fields + raw spans | jshift |
Column appends / RecordBatch | Your Rust (or a thin adapter) |
| Parquet / IPC / Iceberg | arrow / parquet / lakehouse |
Arrow does not force serde. Builders work without it. Serde is optional for JSON decode if you choose that stack instead of jshift.
§Why JsonView can lose to serde on multi-field ingest
#[derive(JsonView)] is optimized for correct open documents and ergonomics,
not for beating serde’s single-pass tokenizer on “decode 5 strings from every
fat object.” Each path field is a selective scan. For stage throughput, use
each_with + find_value (and accept that multi-key single-pass extract is
a future API). See examples/json_parquet_stage_bench.rs and
examples/bronze_land_bench.rs.
§Edge decode vs true raw land
| Prefer edge decode when… | Prefer raw land when… |
|---|---|
| Hot path needs fields before land finishes | Exact wire archive is the only goal |
| Stage is schema-on-write (Parquet rows) at the collector | Silver will reprocess and has CPU |
| You must drop PII / fat tails before storage | Schema is unstable |
Routing / dedupe needs id / account now | Feed is already binary / Arrow |
§Open vs closed documents
| Mode | Meaning | Typical API |
|---|---|---|
| Open | Unmentioned fields unread / preserved on write | path get, write_into, open nested write |
| Closed | Schema-only card | to_schema_bytes, write_view_at_closed, project keep-list |
Open is the product default for API evolution (unknowns stay).
§Enums (tagging styles)
| Style | Wire | Derive |
|---|---|---|
| External (default) | "Ready" / {"Click":{…}} | #[derive(JsonView)] |
| Internal | {"type":"Click","x":1} | #[json(tag = "type")] |
| Adjacent | {"t":"Click","c":{…}} | #[json(tag = "t", content = "c")] |
| Untagged | try variants in order | #[json(untagged)] |
Free functions: read_external,
read_internal, read_adjacent,
read_untagged. Module overview: crate::enums.
§Feature flags (keep core light)
| Feature | When |
|---|---|
default derive | Almost always |
serde | Framework requires Deserialize / gradual migration |
axum | Axum extractors |
parallel | Huge indexed array project |
Core never depends on Arrow/Parquet/Iceberg.
§Examples to run
cargo run --release --example teefury_ingest_bench --features derive
cargo run --release --example bronze_land_bench --features derive
cargo run --release --example json_parquet_stage_bench --features derive
cargo run --release --example multi_json_bench --features derive§Stability note (pre-1.0)
The surfaces above are the intended long-term map. Multi-key single-pass extract and further enum polish may land before 1.0 without changing this “which API for which job” table.