Expand description
jshift — Schema-guided, safe in-place JSON path reader and mutator.
This crate provides a 100% safe Rust engine to selectively read, mutate, upsert,
and delete values inside raw JSON byte buffers (&[u8] and Vec<u8>) without
building a full AST. Path scans return zero-copy slices; mutations resize the
buffer and shift the tail with safe slice rotations.
§What jshift is (and is not)
| Do | Don’t |
|---|---|
| Path index + mutate | Full JSON DOM as the eval spine |
Schema-guided projection (JsonView) / TypedDoc | Multi-format serde (bincode, …) |
| Safe structural side-tables | Promise simdjson Stage-1 crowns |
| Preserve unmentioned fields | Full RFC validator (unless you add one) |
Open documents: paths you don’t name are left unread (and byte-preserved on write). That is a feature for API evolution (same spirit as prost unknown fields).
§Cargo features
| Feature | Default | Purpose |
|---|---|---|
derive | yes | JsonMutatorSchema / JsonView proc-macros |
parallel | no | Rayon-backed parallel list projections on large indexed arrays |
dhat-heap | no | dhat global allocator for examples/alloc_profile (local profiling only) |
Core path engine and structural indexing always compile. Indexing is opt-in at
the call site (IndexedDocument, read_from_indexed); never taxed on default finds.
Parallel project is opt-in (project_parallel / project_indexed_parallel); use
project_indexed_auto / project_parallel_auto to enable Rayon only when
plan_prefers_parallel says the plan is CPU-heavy. Streaming cards:
project_each / project_jsonl_write (no giant output array).
jshift = { version = "0.5", features = ["derive"] }
# core only (no proc-macros):
jshift = { version = "0.5", default-features = false }
# bulk array project with rayon:
jshift = { version = "0.5", features = ["parallel"] }Measurement: Criterion (benches/), RSS (scripts/measure_rss.sh), dhat/heaptrack
(examples/alloc_profile). Large fixtures: scripts/fetch_teefury.sh +
scripts/build_large_catalog.sh → gitignored benches/data/large.json.
CI verifies both default and --no-default-features builds.
§Features
- Zero-copy reads: Find values as slices into the raw buffer.
- In-place mutations: Safe byte-shifting (including resize) via slice rotations.
JsonViewtrait: one protocol surface for typed projections of bytes.- Macro-generated schemas:
#[derive(JsonView)]/JsonMutatorSchema. - Typed documents:
JsonDoc+TypedDoc/TypedDocRef,ViewList/ValueList,RawJson, exclusiveTypedMutator. - Build without Value:
ObjectBuilder/ArrayBuilder/JsonWriter(in-place nest);TypedDoc::from_view. - Batch mutate:
MutateOp/BatchPlan/apply_ops;rename_key,merge_object_shallow. - Validate:
require_paths,deny_unknown_keys,validate_closed/validate_open(no DOM). - Shared buffers:
SharedDocument(Arc<[u8]>) for read-heavy fan-out. - JSONL helpers:
json_lines,jsonl_docs,read_jsonl,write_jsonl_views; array→NDJSON cards viaproject_jsonl_write. - Field projection:
project/project_paths/project_jmespath/ProjectPlan(keep-list + JMESPath subset → new JSON); streamingproject_eachfor per-row cards without a giant output array. - Projection estimates:
estimate_projected_len/projected_len. - Structural array indexes:
IndexedDocumentside-tables for large arrays. - Parallel auto-pick:
plan_prefers_parallel+project_indexed_auto.
§Quick Start
use jshift::{find_value, mutate_value, parse_path};
let mut json = b"{\"user\": \"farmer\", \"score\": 9.5}".to_vec();
let path = parse_path("score");
// Read value
let score_bytes = find_value(&json, &path).unwrap();
assert_eq!(score_bytes, b"9.5");
// Mutate in-place
mutate_value(&mut json, &path, b"10.0").unwrap();
assert_eq!(json, b"{\"user\": \"farmer\", \"score\": 10.0}".to_vec());§TypedDoc: typed fields without Value
use jshift::{JsonDoc, TypedDoc};
let mut doc = TypedDoc::from_slice(br#"{"status":"ok","id":7,"items":[{"n":1}],"extra":true}"#);
assert_eq!(doc.get::<u64>("id").unwrap(), 7);
assert_eq!(doc.get_str("status").unwrap(), "ok");
// sparse array field stream (relative path parsed once)
assert_eq!(doc.collect_each_get::<u64>("items", "n").unwrap(), vec![1]);
doc.mutate().set("status", "accepted").unwrap();
assert_eq!(doc.get_str("status").unwrap(), "accepted");
assert!(doc.contains("extra").unwrap());§JsonView: typed projections (open partial structs)
use jshift::{JsonView, JsonMutatorSchema};
#[derive(JsonMutatorSchema)]
struct ListingCard {
#[json(path = "id")]
id: u64,
#[json(path = "title")]
title: String,
// intentionally no variants / images: partial view
}
fn ingest<T: JsonView>(buf: &[u8]) -> Result<T, jshift::Error> {
T::read_from(buf)
}
let json = br#"{"id":7,"title":"Hat","images":[1,2,3],"variants":[]}"#;
let card: ListingCard = ingest(json).unwrap();
assert_eq!(card.id, 7);
assert_eq!(card.title, "Hat");
// write_into only touches named paths; other keys stay put
let mut buf = json.to_vec();
card.write_into(&mut buf).unwrap();
assert!(buf.windows(8).any(|w| w == b"\"images\""));§Project: keep-list / JMESPath subset → smaller JSON
use jshift::{project_jmespath, project_paths, ProjectPlan, ProjectStyle};
let json = br#"{"id":7,"title":"Hat","images":[1,2,3],"meta":{"x":1}}"#;
let out = project_paths(json, &["id", "title", "meta.x"]).unwrap();
assert_eq!(out, br#"{"id":7,"title":"Hat","meta":{"x":1}}"#);
// Keep-list wildcards preserve ancestor keys:
let catalog = br#"{"products":[{"sku":"A","blob":1},{"sku":"B","blob":2}]}"#;
assert_eq!(
project_paths(catalog, &["products[].sku"]).unwrap(),
br#"{"products":[{"sku":"A"},{"sku":"B"}]}"#
);
// JMESPath subset: multi-select hash + rename (result is the projected array):
assert_eq!(
project_jmespath(catalog, "products[*].{sku: sku}").unwrap(),
br#"[{"sku":"A"},{"sku":"B"}]"#
);
let pretty = ProjectPlan::from_paths(&["id"]).unwrap().style(ProjectStyle::Pretty { indent: 2 });
let _ = jshift::project(json, &pretty).unwrap();§High-Impact Real-World Use Case: LLM Dataset Processing (JSONL)
In AI training pipelines (e.g., LoRA finetuning), datasets are stored as JSONL files. You can inspect token lengths and mark records as skipped or cleaned in-place:
use jshift::JsonMutatorSchema;
#[derive(JsonMutatorSchema)]
struct TrainingRecord {
#[json(path = "tokens")]
tokens: usize,
#[json(path = "status")]
status: String,
}
let mut line = b"{\"instruction\": \"Translate...\", \"tokens\": 1024, \"status\": \"pending\"}".to_vec();
// Parse selectively
let record = TrainingRecord::read_from_json(&line).unwrap();
// Skip long contexts in-place!
if record.tokens > 512 {
let mut mutator = TrainingRecord::mutator(&mut line);
mutator.set_status("skipped").unwrap();
}
assert_eq!(
line,
b"{\"instruction\": \"Translate...\", \"tokens\": 1024, \"status\": \"skipped\"}".to_vec()
);Structs§
- Array
Builder - Fluent builder for a JSON array (
[...]). - Array
Elems - Fallible iterator over JSON array element spans.
- Array
Index - Side-table for one array value: start offset of each element.
- Batch
Plan - Fluent collector for
MutateOps, thencommitonto a buffer. - Counting
Sink - Counts bytes that would be written (exact projected length without retaining output).
- Hash
Field - One field in a multi-select hash (
output_key: expr). - Indexed
Document - A JSON buffer plus structural / array / object indexes for path-accelerated reads.
- Indexed
Value List ValueListafterValueList::index— O(1) element access.- Indexed
View List ViewListafterViewList::index— O(1) element access by index.- Json
Lines - Iterator over non-empty JSONL lines as byte slices.
- Json
Writer - Imperative JSON encoder into one growable buffer.
- Object
Builder - Fluent builder for a JSON object (
{...}). - Object
Entries - Fallible iterator over object key/value spans (DynObject cursor).
- Object
Entry - One object member: on-wire key content + raw value span.
- Object
KeyIndex - Map of object keys (on-wire bytes between quotes) → value span.
- Object
Select - Object field selection for subset projection.
- Path
- Owned, reusable JSON path.
- Project
Plan - Compiled projection plan: selection AST + formatting + missing policy.
- RawJson
- Owned raw JSON value bytes (object, array, string, number, bool, or null).
- Shared
Document - Cheaply cloneable, read-only JSON buffer.
- Structural
Index - Sorted absolute offsets of structural characters outside JSON strings:
{ } [ ] : , - Transform
Pipeline - Ordered list of
Transformsteps. - Typed
Doc - Owned JSON buffer with typed path access and exclusive in-place mutation.
- Typed
DocRef - Borrowed, read-only typed document over
&[u8]. - Typed
Mutator - Exclusive mutator for a
TypedDoc. - Value
List - Cursor over a JSON array of values decoded via
FromJsonSlice. - Value
List Iter - Iterator yielding
Result<T, Error>for each primitive/array element. - View
List - Cursor over a JSON array of objects decoded as
T: JsonView. - View
List Iter - Iterator yielding
Result<T, Error>for each array element. - Write
Sink - Streams projected bytes to
W(no full-output buffer).
Enums§
- Array
Select - Array projection strategy.
- CmpOp
- Comparison operators (JMESPath).
- Error
- Errors returned by scanning and mutating operations.
- Json
Type - Type-check a path as a JSON string / number / bool / object / array without full decode.
- Missing
Policy - Behavior when a selected path is missing in the input.
- Mutate
Op - One mutation in a batch plan.
- Owned
Path Segment - Owned path segment (see
Path). - Path
Segment - Represents a segment in a JSON path (either a string key or a zero-indexed array index).
- Presence
- Whether a path is absent, JSON
null, or a non-null value. - Project
Path Segment - One step of a keep-list path (plus wildcards / slices).
- Project
Style - How projected structure is formatted.
- Root
Kind - Kind of a JSON value at the document root (or any single-value span).
- Select
Expr - Selection / expression node over a JSON value span.
- Transform
- One transform step in a pipeline.
Traits§
- From
Json Slice - Trait implemented by types that can be parsed directly from a raw JSON byte slice.
- JsonDoc
- Shared typed read surface over any JSON byte buffer.
- Json
View - A Rust type that is a projection of JSON bytes along named paths.
- ToJson
Bytes - Trait implemented by types that can be serialized directly into a JSON byte representation.
Functions§
- append_
to_ array - Appends a new value to the end of a JSON array located at the specified path.
- apply_
ops - Apply
opsin order to a raw buffer. - array_
from_ iter - Convenience: build an array from values.
- array_
len - Returns the number of elements in the array located at the specified path.
- build_
array - Build a root array by driving a
JsonWritercallback. - build_
array_ index - Build an
ArrayIndexfor the array atpath. - build_
object - Build a root object by driving a
JsonWritercallback. - build_
object_ key_ index - Build an
ObjectKeyIndexfor the object atpath. - build_
structural_ index - Build a document-wide Stage-1 structural index (safe).
- delete_
index - Deletes an element from a JSON array located at the specified path by its index.
- delete_
key - Deletes a key-value pair from a JSON object located at the specified path.
- deny_
unknown_ keys - Closed-object check: every key on the object at
object_pathmust be inallowed(logical unescaped keys for unescaped on-wire keys). - deny_
unknown_ keys_ at - Like
deny_unknown_keysbutobject_pathis a string path (""or omit root). - escape_
json_ key - Escape
sas it would appear inside a JSON string (no surrounding quotes). - escape_
json_ string - Serialize
sas a JSON string literal (including surrounding quotes). - estimate_
projected_ len - Ballpark capacity hint for flat keep-lists (not exact projector output).
- estimate_
values_ len - Sum of on-wire value lengths for
paths. - find_
value - Locates a JSON value byte-slice within a raw JSON buffer by its path.
- for_
each_ jsonl_ doc - Map each line as
TypedDocRefwith a fallible callback. - for_
each_ jsonl_ line - Map JSONL lines with a fallible callback (stream; no intermediate
Vec). - from_
jshift_ bytes - Happy-path alias: bytes →
T: JsonView(roadmap adoption surface). - from_
json_ string - Parse a JSON string literal (including surrounding quotes) into an unescaped
String. - insert_
array_ element - Insert
new_elementinto the array atpathso it becomes the element atindex. - json_
lines - Iterate non-empty lines in a JSONL / NDJSON buffer (zero-copy slices).
- jsonl_
docs - Zero-copy
TypedDocRefper non-empty JSONL line. - jsonl_
docs_ owned - Owned
TypedDocper line (copies each line into aVec<u8>). - merge_
object_ shallow - Shallow-merge
patchobject members into the object atpath. - mutate_
value - Mutates a JSON value in-place inside a
Vec<u8>buffer by its path. - mutate_
value_ checked - Like
mutate_value, but requiresnew_valueto parse as exactly one complete JSON value (object, array, string, number,true/false/null) with no trailing junk after optional whitespace. - object_
from_ iter - Convenience: build an object from
(key, value)pairs (call order preserved). - parse_
jmespath_ expr - Parse a JMESPath expression into a
SelectExpr. - parse_
path - Parses a dot-and-bracket notation path string into zero-copy path segments.
- parse_
project_ path - Parse projection path segments including signed slices and wildcards.
- plan_
object_ fields - Build a
ProjectPlanforproject_object_fields(array of objects → multi-select hash). - plan_
prefers_ parallel - Heuristic: prefer Rayon list emit when per-element work is likely CPU-bound.
- prepend_
to_ array - Insert
new_elementas the first element of the array atpath. - project
- Project
jsonaccording toplan, returning a new buffer. - project_
as_ view - Project
jsondown toT’s keep-list, then decode asT. - project_
auto_ indexed - Build structural + plan array indexes, then project (one-shot convenience).
- project_
each - Project each element of a list projection plan as a separate card.
- project_
each_ indexed - Like
project_each, using side-tables fromdocwhen the array path is indexed. - project_
indexed - Project using a pre-built
IndexedDocument. - project_
indexed_ auto - Project with side-tables, enabling parallel
[*]only whenplan_prefers_parallel. - project_
indexed_ prepare - Ensure
dochas side-tables for the plan’s array paths, then project. - project_
into - Project into an existing buffer (appends).
- project_
jmespath - Convenience:
ProjectPlan::from_jmespaththenproject. - project_
jsonl_ write - Write each list-projection card as one NDJSON line (JSON object + newline) to
w. - project_
jsonl_ write_ indexed - Indexed NDJSON list projection (reuse side-tables).
- project_
object_ fields - Generic “thin cards” helper: project each object in an array down to named fields.
- project_
object_ fields_ each - Thin-card variant: project named fields on each object in
array_path, one callback per element (no giant result array). - project_
object_ fields_ each_ indexed - Indexed thin-card each-callback (prefer when reusing an
IndexedDocument). - project_
object_ fields_ jsonl_ write - NDJSON thin cards: each object in
array_path→ one line with selected fields. - project_
parallel_ auto - Build plan indexes, then
project_indexed_auto(seq vs parallel by heuristic). - project_
paths - Convenience:
ProjectPlan::from_pathsthenproject. - project_
write - Stream projection into any
Writewithout buffering the full output. - projected_
len - Exact projected byte length without retaining the output (counting sink).
- read_
jsonl - Map each non-empty JSONL line through a
JsonViewreader. - read_
jsonl_ indexed - Map each line with schema-guided indexing (rebuild index per line).
- read_
line_ indexed - Index a single JSONL line (array paths), then read a view.
- read_
view - Shared helper: read any
JsonViewfrom bytes. - rename_
key - Rename an object member key in place (value bytes unchanged).
- require_
paths - Ensure every path exists (any non-missing value, including JSON
null). - require_
paths_ non_ null - Ensure every path exists and is not JSON
null. - require_
type - Require
pathto have typeexpected. - select_
from_ project_ path - Build
SelectExprfrom a keep-list path. - set_at
- Convenience: build from a pre-parsed
Pathset (avoids re-tokenize in hot loops). - static_
array_ prefixes_ from_ path - Extract static array path prefixes from a field path for derive auto-index.
- try_
parse_ path - Strict path parser. Returns
Error::InvalidPathfor malformed path syntax instead of silently dropping segments. - type_at
- Sniff the JSON type at
path. - upsert_
at - upsert_
at_ path - Upsert
new_valueatpath, creating missing object parents as{}. - upsert_
object_ key - Upserts (inserts or updates) a key-value pair in a JSON object located at the specified path.
- validate_
closed - Closed mode: required paths present and no unknown keys on the root object.
- validate_
open - Open mode: only check that required paths exist; unknowns are ignored.
- write_
json_ string - Append a JSON string literal (including surrounding quotes) for
sintoout. - write_
json_ string_ content - Append the escaped form of
s(content only, no surrounding quotes) intoout. - write_
jsonl_ docs - Write each document’s bytes as one NDJSON line.
- write_
jsonl_ line - Write one JSON value as a JSONL line (
raw+\n). - write_
jsonl_ views - Write each view as one NDJSON line via
TypedDoc::from_view. - write_
view - Shared helper: write any
JsonViewinto a buffer.
Derive Macros§
- Json
Mutator Schema - Derive typed JSON path readers/mutators and implement [
jshift::JsonView]. - Json
View - Alias for
JsonMutatorSchema: a Rust type is a projection of JSON bytes.