Skip to main content

Module api_guide

Module api_guide 

Source
Expand description

Product map: which API for which job (ingest, views, project, enums, bridges).

§API guide — when to use what

jshift exposes several surfaces that look similar (JsonView, path gets, project, multi-page streams, optional serde/axum bridges). Picking the wrong one for the job is the main way to get surprising benches (e.g. serde beating a JsonView card on multi-field ingest).

This guide is the product map. Prefer it over inventing a parallel style.

§One-line product

Schema-guided JSON on raw bytes: sparse path read, open in-place mutate, project/stream — without a serde_json::Value tree as the spine.

§When to use which API

JobPreferAvoid for this job
Hot path: 1–few fields (id, seq, flag)JsonDoc::get / each_get / find_valueFull [JsonView] card, Value
Ingest / stage tables (cols + optional raw_json)each_with + find_value (static PathSegments); raw = element spanJsonView multi-string card as the hot loop
Domain object (handler, business logic, open mutate)#[derive(JsonView)] + read_from / write_intoHand-rolled find_value soup for 20 fields
Thin JSON out (drop variants/images)project / project_jmespath / ProjectPlanDeserialize full tree then re-serialize
Archive bronze (exact wire bytes)Write the buffer; hold TypedDoc if you need paths laterParsing into Arrow “for storage” if you only need raw
Paginated / multi-file / JSONLfor_each_page_items, for_each_jsonl_multi, JsonlDecoderConcatenate pages into one mega-buffer
HTTP body → view (axum)feature axum: JshiftJson / JshiftDocJson<Value> then convert
Must satisfy Deserializefeature serde: AsJshiftView — see serde bridge guideImplementing Deserialize on every jshift type by default
Externally tagged enums#[derive(JsonView)] default, or read_external_*Value match on tags
Internally / adjacent / untagged enums#[json(tag = …)] / content / untagged (0.7+) — see crate::enumsAssuming only external exists

§Ingest pattern (Arrow / Parquet hand-off)

jshift does not build Arrow rows. Your loop does:

ⓘ
use jshift::{find_value, FromJsonSlice, JsonDoc, PathSegment, TypedDocRef};

const P_ID: &[PathSegment] = &[PathSegment::Key("id")];
const P_TITLE: &[PathSegment] = &[PathSegment::Key("title")];

let doc = TypedDocRef::from_slice(page_json);
doc.each_with("products", |elem| {
    let id = u64::from_json_slice(find_value(elem, P_ID)?)?;
    let title = String::from_json_slice(find_value(elem, P_TITLE)?)?;
    let raw_json = std::str::from_utf8(elem)?; // full product span
    // id_col.append_value(id);
    // title_col.append_value(&title);
    // raw_col.append_value(raw_json);
    let _ = (id, title, raw_json);
    Ok(())
})?;
// finish builders → RecordBatch → Parquet (arrow/parquet crates, not jshift)
LayerOwner
Nested JSON on the wireNetwork / files
Sparse fields + raw spansjshift
Column appends / RecordBatchYour Rust (or a thin adapter)
Parquet / IPC / Icebergarrow / parquet / lakehouse

Arrow does not force serde. Builders work without it. Serde is optional for JSON decode if you choose that stack instead of jshift.

§Why JsonView can lose to serde on multi-field ingest

#[derive(JsonView)] is optimized for correct open documents and ergonomics, not for beating serde’s single-pass tokenizer on “decode 5 strings from every fat object.” Each path field is a selective scan. For stage throughput, use each_with + find_value (and accept that multi-key single-pass extract is a future API). See examples/json_parquet_stage_bench.rs and examples/bronze_land_bench.rs.

§Edge decode vs true raw land

Prefer edge decode when…Prefer raw land when…
Hot path needs fields before land finishesExact wire archive is the only goal
Stage is schema-on-write (Parquet rows) at the collectorSilver will reprocess and has CPU
You must drop PII / fat tails before storageSchema is unstable
Routing / dedupe needs id / account nowFeed is already binary / Arrow

§Open vs closed documents

ModeMeaningTypical API
OpenUnmentioned fields unread / preserved on writepath get, write_into, open nested write
ClosedSchema-only cardto_schema_bytes, write_view_at_closed, project keep-list

Open is the product default for API evolution (unknowns stay).

§Enums (tagging styles)

StyleWireDerive
External (default)"Ready" / {"Click":{…}}#[derive(JsonView)]
Internal{"type":"Click","x":1}#[json(tag = "type")]
Adjacent{"t":"Click","c":{…}}#[json(tag = "t", content = "c")]
Untaggedtry variants in order#[json(untagged)]

Free functions: read_external, read_internal, read_adjacent, read_untagged. Module overview: crate::enums.

§Feature flags (keep core light)

FeatureWhen
default deriveAlmost always
serdeFramework requires Deserialize / gradual migration
axumAxum extractors
parallelHuge indexed array project

Core never depends on Arrow/Parquet/Iceberg.

§Examples to run

cargo run --release --example teefury_ingest_bench --features derive
cargo run --release --example bronze_land_bench --features derive
cargo run --release --example json_parquet_stage_bench --features derive
cargo run --release --example multi_json_bench --features derive

§Stability note (pre-1.0)

The surfaces above are the intended long-term map. Multi-key single-pass extract and further enum polish may land before 1.0 without changing this “which API for which job” table.