1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
//! # API guide — when to use what
//!
//! jshift exposes several surfaces that look similar (`JsonView`, path gets,
//! project, multi-page streams, optional serde/axum bridges). **Picking the
//! wrong one for the job** is the main way to get surprising benches (e.g.
//! serde beating a `JsonView` card on multi-field ingest).
//!
//! This guide is the product map. Prefer it over inventing a parallel style.
//!
//! # One-line product
//!
//! **Schema-guided JSON on raw bytes:** sparse path read, open in-place mutate,
//! project/stream — without a `serde_json::Value` tree as the spine.
//!
//! # When to use which API
//!
//! | Job | Prefer | Avoid for this job |
//! | --- | --- | --- |
//! | **Hot path:** 1–few fields (`id`, `seq`, flag) | [`JsonDoc::get`] / [`each_get`](JsonDoc::each_get) / [`find_value`] | Full [`JsonView`] card, `Value` |
//! | **Ingest / stage tables** (cols + optional `raw_json`) | [`each_with`](JsonDoc::each_with) + [`find_value`] (static [`PathSegment`]s); raw = element span | `JsonView` multi-string card as the hot loop |
//! | **Domain object** (handler, business logic, open mutate) | `#[derive(JsonView)]` + [`read_from`](JsonView::read_from) / [`write_into`](JsonView::write_into) | Hand-rolled `find_value` soup for 20 fields |
//! | **Thin JSON out** (drop variants/images) | [`project`] / [`project_jmespath`] / [`ProjectPlan`] | Deserialize full tree then re-serialize |
//! | **Archive bronze** (exact wire bytes) | Write the buffer; hold [`TypedDoc`] if you need paths later | Parsing into Arrow “for storage” if you only need raw |
//! | **Paginated / multi-file / JSONL** | [`for_each_page_items`], [`for_each_jsonl_multi`], [`JsonlDecoder`] | Concatenate pages into one mega-buffer |
//! | **HTTP body → view** (axum) | feature `axum`: [`JshiftJson`](crate::JshiftJson) / [`JshiftDoc`](crate::JshiftDoc) | `Json<Value>` then convert |
//! | **Must satisfy `Deserialize`** | feature `serde`: [`AsJshiftView`](crate::AsJshiftView) — see [serde bridge guide](crate::serde_bridge_guide) | Implementing `Deserialize` on every jshift type by default |
//! | **Externally tagged enums** | `#[derive(JsonView)]` default, or [`read_external_*`](crate::read_external) | `Value` match on tags |
//! | **Internally / adjacent / untagged enums** | `#[json(tag = …)]` / `content` / `untagged` (0.7+) — see [`crate::enums`] | Assuming only external exists |
//!
//! [`JsonDoc::get`]: crate::JsonDoc::get
//! [`find_value`]: crate::find_value
//! [`PathSegment`]: crate::PathSegment
//! [`project`]: crate::project
//! [`project_jmespath`]: crate::project_jmespath
//! [`ProjectPlan`]: crate::ProjectPlan
//! [`TypedDoc`]: crate::TypedDoc
//! [`for_each_page_items`]: crate::for_each_page_items
//! [`for_each_jsonl_multi`]: crate::for_each_jsonl_multi
//! [`JsonlDecoder`]: crate::JsonlDecoder
//!
//! # Ingest pattern (Arrow / Parquet hand-off)
//!
//! jshift does **not** build Arrow rows. Your loop does:
//!
//! ```rust,ignore
//! use jshift::{find_value, FromJsonSlice, JsonDoc, PathSegment, TypedDocRef};
//!
//! const P_ID: &[PathSegment] = &[PathSegment::Key("id")];
//! const P_TITLE: &[PathSegment] = &[PathSegment::Key("title")];
//!
//! let doc = TypedDocRef::from_slice(page_json);
//! doc.each_with("products", |elem| {
//! let id = u64::from_json_slice(find_value(elem, P_ID)?)?;
//! let title = String::from_json_slice(find_value(elem, P_TITLE)?)?;
//! let raw_json = std::str::from_utf8(elem)?; // full product span
//! // id_col.append_value(id);
//! // title_col.append_value(&title);
//! // raw_col.append_value(raw_json);
//! let _ = (id, title, raw_json);
//! Ok(())
//! })?;
//! // finish builders → RecordBatch → Parquet (arrow/parquet crates, not jshift)
//! ```
//!
//! | Layer | Owner |
//! | --- | --- |
//! | Nested JSON on the wire | Network / files |
//! | Sparse fields + raw spans | **jshift** |
//! | Column appends / `RecordBatch` | **Your Rust** (or a thin adapter) |
//! | Parquet / IPC / Iceberg | **arrow / parquet / lakehouse** |
//!
//! **Arrow does not force serde.** Builders work without it. Serde is optional for
//! JSON decode if you choose that stack instead of jshift.
//!
//! # Why `JsonView` can lose to serde on multi-field ingest
//!
//! `#[derive(JsonView)]` is optimized for **correct open documents and ergonomics**,
//! not for beating serde’s single-pass tokenizer on “decode 5 strings from every
//! fat object.” Each path field is a selective scan. For stage throughput, use
//! **`each_with` + `find_value`** (and accept that multi-key single-pass extract is
//! a future API). See `examples/json_parquet_stage_bench.rs` and
//! `examples/bronze_land_bench.rs`.
//!
//! # Edge decode vs true raw land
//!
//! | Prefer **edge decode** when… | Prefer **raw land** when… |
//! | --- | --- |
//! | Hot path needs fields before land finishes | Exact wire archive is the only goal |
//! | Stage is schema-on-write (Parquet rows) at the collector | Silver will reprocess and has CPU |
//! | You must drop PII / fat tails before storage | Schema is unstable |
//! | Routing / dedupe needs `id` / `account` now | Feed is already binary / Arrow |
//!
//! # Open vs closed documents
//!
//! | Mode | Meaning | Typical API |
//! | --- | --- | --- |
//! | **Open** | Unmentioned fields unread / preserved on write | path get, `write_into`, open nested write |
//! | **Closed** | Schema-only card | `to_schema_bytes`, `write_view_at_closed`, project keep-list |
//!
//! Open is the product default for API evolution (unknowns stay).
//!
//! # Enums (tagging styles)
//!
//! | Style | Wire | Derive |
//! | --- | --- | --- |
//! | **External** (default) | `"Ready"` / `{"Click":{…}}` | `#[derive(JsonView)]` |
//! | **Internal** | `{"type":"Click","x":1}` | `#[json(tag = "type")]` |
//! | **Adjacent** | `{"t":"Click","c":{…}}` | `#[json(tag = "t", content = "c")]` |
//! | **Untagged** | try variants in order | `#[json(untagged)]` |
//!
//! Free functions: [`read_external`](crate::read_external),
//! [`read_internal`](crate::read_internal), [`read_adjacent`](crate::read_adjacent),
//! [`read_untagged`](crate::read_untagged). Module overview: [`crate::enums`].
//!
//! # Feature flags (keep core light)
//!
//! | Feature | When |
//! | --- | --- |
//! | default `derive` | Almost always |
//! | `serde` | Framework requires `Deserialize` / gradual migration |
//! | `axum` | Axum extractors |
//! | `parallel` | Huge indexed array project |
//!
//! Core never depends on Arrow/Parquet/Iceberg.
//!
//! # Examples to run
//!
//! ```text
//! cargo run --release --example teefury_ingest_bench --features derive
//! cargo run --release --example bronze_land_bench --features derive
//! cargo run --release --example json_parquet_stage_bench --features derive
//! cargo run --release --example multi_json_bench --features derive
//! ```
//!
//! # Stability note (pre-1.0)
//!
//! The surfaces above are the intended long-term map. Multi-key single-pass
//! extract and further enum polish may land before 1.0 without changing this
//! “which API for which job” table.