Skip to main content

Crate jshift

Crate jshift 

Source
Expand description

jshift — Schema-guided, safe in-place JSON path reader and mutator.

This crate provides a 100% safe Rust engine to selectively read, mutate, upsert, and delete values inside raw JSON byte buffers (&[u8] and Vec<u8>) without building a full AST. Path scans return zero-copy slices; mutations resize the buffer and shift the tail with safe slice rotations.

§What jshift is (and is not)

DoDon’t
Path index + mutateFull JSON DOM as the eval spine
Schema-guided projection (JsonView) / TypedDocMulti-format serde (bincode, …)
Safe structural side-tablesPromise simdjson Stage-1 crowns
Preserve unmentioned fieldsFull RFC validator (unless you add one)

Open documents: paths you don’t name are left unread (and byte-preserved on write). That is a feature for API evolution (same spirit as prost unknown fields).

§Cargo features

FeatureDefaultPurpose
deriveyesJsonMutatorSchema / JsonView proc-macros
parallelnoRayon-backed parallel list projections on large indexed arrays
dhat-heapnodhat global allocator for examples/alloc_profile (local profiling only)

Core path engine and structural indexing always compile. Indexing is opt-in at the call site (IndexedDocument, read_from_indexed); never taxed on default finds. Parallel project is opt-in (project_parallel / project_indexed_parallel); use project_indexed_auto / project_parallel_auto to enable Rayon only when plan_prefers_parallel says the plan is CPU-heavy. Streaming cards: project_each / project_jsonl_write (no giant output array).

jshift = { version = "0.5", features = ["derive"] }
# core only (no proc-macros):
jshift = { version = "0.5", default-features = false }
# bulk array project with rayon:
jshift = { version = "0.5", features = ["parallel"] }

Measurement: Criterion (benches/), RSS (scripts/measure_rss.sh), dhat/heaptrack (examples/alloc_profile). Large fixtures: scripts/fetch_teefury.sh + scripts/build_large_catalog.sh → gitignored benches/data/large.json.

CI verifies both default and --no-default-features builds.

§Features

§Quick Start

use jshift::{find_value, mutate_value, parse_path};

let mut json = b"{\"user\": \"farmer\", \"score\": 9.5}".to_vec();
let path = parse_path("score");

// Read value
let score_bytes = find_value(&json, &path).unwrap();
assert_eq!(score_bytes, b"9.5");

// Mutate in-place
mutate_value(&mut json, &path, b"10.0").unwrap();
assert_eq!(json, b"{\"user\": \"farmer\", \"score\": 10.0}".to_vec());

§TypedDoc: typed fields without Value

use jshift::{JsonDoc, TypedDoc};

let mut doc = TypedDoc::from_slice(br#"{"status":"ok","id":7,"items":[{"n":1}],"extra":true}"#);
assert_eq!(doc.get::<u64>("id").unwrap(), 7);
assert_eq!(doc.get_str("status").unwrap(), "ok");
// sparse array field stream (relative path parsed once)
assert_eq!(doc.collect_each_get::<u64>("items", "n").unwrap(), vec![1]);

doc.mutate().set("status", "accepted").unwrap();
assert_eq!(doc.get_str("status").unwrap(), "accepted");
assert!(doc.contains("extra").unwrap());

§JsonView: typed projections (open partial structs)

use jshift::{JsonView, JsonMutatorSchema};

#[derive(JsonMutatorSchema)]
struct ListingCard {
    #[json(path = "id")]
    id: u64,
    #[json(path = "title")]
    title: String,
    // intentionally no variants / images: partial view
}

fn ingest<T: JsonView>(buf: &[u8]) -> Result<T, jshift::Error> {
    T::read_from(buf)
}

let json = br#"{"id":7,"title":"Hat","images":[1,2,3],"variants":[]}"#;
let card: ListingCard = ingest(json).unwrap();
assert_eq!(card.id, 7);
assert_eq!(card.title, "Hat");

// write_into only touches named paths; other keys stay put
let mut buf = json.to_vec();
card.write_into(&mut buf).unwrap();
assert!(buf.windows(8).any(|w| w == b"\"images\""));

§Project: keep-list / JMESPath subset → smaller JSON

use jshift::{project_jmespath, project_paths, ProjectPlan, ProjectStyle};

let json = br#"{"id":7,"title":"Hat","images":[1,2,3],"meta":{"x":1}}"#;
let out = project_paths(json, &["id", "title", "meta.x"]).unwrap();
assert_eq!(out, br#"{"id":7,"title":"Hat","meta":{"x":1}}"#);

// Keep-list wildcards preserve ancestor keys:
let catalog = br#"{"products":[{"sku":"A","blob":1},{"sku":"B","blob":2}]}"#;
assert_eq!(
    project_paths(catalog, &["products[].sku"]).unwrap(),
    br#"{"products":[{"sku":"A"},{"sku":"B"}]}"#
);

// JMESPath subset: multi-select hash + rename (result is the projected array):
assert_eq!(
    project_jmespath(catalog, "products[*].{sku: sku}").unwrap(),
    br#"[{"sku":"A"},{"sku":"B"}]"#
);

let pretty = ProjectPlan::from_paths(&["id"]).unwrap().style(ProjectStyle::Pretty { indent: 2 });
let _ = jshift::project(json, &pretty).unwrap();

§High-Impact Real-World Use Case: LLM Dataset Processing (JSONL)

In AI training pipelines (e.g., LoRA finetuning), datasets are stored as JSONL files. You can inspect token lengths and mark records as skipped or cleaned in-place:

use jshift::JsonMutatorSchema;

#[derive(JsonMutatorSchema)]
struct TrainingRecord {
    #[json(path = "tokens")]
    tokens: usize,
    #[json(path = "status")]
    status: String,
}

let mut line = b"{\"instruction\": \"Translate...\", \"tokens\": 1024, \"status\": \"pending\"}".to_vec();

// Parse selectively
let record = TrainingRecord::read_from_json(&line).unwrap();

// Skip long contexts in-place!
if record.tokens > 512 {
    let mut mutator = TrainingRecord::mutator(&mut line);
    mutator.set_status("skipped").unwrap();
}

assert_eq!(
    line,
    b"{\"instruction\": \"Translate...\", \"tokens\": 1024, \"status\": \"skipped\"}".to_vec()
);

Structs§

ArrayBuilder
Fluent builder for a JSON array ([...]).
ArrayElems
Fallible iterator over JSON array element spans.
ArrayIndex
Side-table for one array value: start offset of each element.
BatchPlan
Fluent collector for MutateOps, then commit onto a buffer.
CountingSink
Counts bytes that would be written (exact projected length without retaining output).
HashField
One field in a multi-select hash (output_key: expr).
IndexedDocument
A JSON buffer plus structural / array / object indexes for path-accelerated reads.
IndexedValueList
ValueList after ValueList::index — O(1) element access.
IndexedViewList
ViewList after ViewList::index — O(1) element access by index.
JsonLines
Iterator over non-empty JSONL lines as byte slices.
JsonWriter
Imperative JSON encoder into one growable buffer.
ObjectBuilder
Fluent builder for a JSON object ({...}).
ObjectEntries
Fallible iterator over object key/value spans (DynObject cursor).
ObjectEntry
One object member: on-wire key content + raw value span.
ObjectKeyIndex
Map of object keys (on-wire bytes between quotes) → value span.
ObjectSelect
Object field selection for subset projection.
Path
Owned, reusable JSON path.
ProjectPlan
Compiled projection plan: selection AST + formatting + missing policy.
RawJson
Owned raw JSON value bytes (object, array, string, number, bool, or null).
SharedDocument
Cheaply cloneable, read-only JSON buffer.
StructuralIndex
Sorted absolute offsets of structural characters outside JSON strings: { } [ ] : ,
TransformPipeline
Ordered list of Transform steps.
TypedDoc
Owned JSON buffer with typed path access and exclusive in-place mutation.
TypedDocRef
Borrowed, read-only typed document over &[u8].
TypedMutator
Exclusive mutator for a TypedDoc.
ValueList
Cursor over a JSON array of values decoded via FromJsonSlice.
ValueListIter
Iterator yielding Result<T, Error> for each primitive/array element.
ViewList
Cursor over a JSON array of objects decoded as T: JsonView.
ViewListIter
Iterator yielding Result<T, Error> for each array element.
WriteSink
Streams projected bytes to W (no full-output buffer).

Enums§

ArraySelect
Array projection strategy.
CmpOp
Comparison operators (JMESPath).
Error
Errors returned by scanning and mutating operations.
JsonType
Type-check a path as a JSON string / number / bool / object / array without full decode.
MissingPolicy
Behavior when a selected path is missing in the input.
MutateOp
One mutation in a batch plan.
OwnedPathSegment
Owned path segment (see Path).
PathSegment
Represents a segment in a JSON path (either a string key or a zero-indexed array index).
Presence
Whether a path is absent, JSON null, or a non-null value.
ProjectPathSegment
One step of a keep-list path (plus wildcards / slices).
ProjectStyle
How projected structure is formatted.
RootKind
Kind of a JSON value at the document root (or any single-value span).
SelectExpr
Selection / expression node over a JSON value span.
Transform
One transform step in a pipeline.

Traits§

FromJsonSlice
Trait implemented by types that can be parsed directly from a raw JSON byte slice.
JsonDoc
Shared typed read surface over any JSON byte buffer.
JsonView
A Rust type that is a projection of JSON bytes along named paths.
ToJsonBytes
Trait implemented by types that can be serialized directly into a JSON byte representation.

Functions§

append_to_array
Appends a new value to the end of a JSON array located at the specified path.
apply_ops
Apply ops in order to a raw buffer.
array_from_iter
Convenience: build an array from values.
array_len
Returns the number of elements in the array located at the specified path.
build_array
Build a root array by driving a JsonWriter callback.
build_array_index
Build an ArrayIndex for the array at path.
build_object
Build a root object by driving a JsonWriter callback.
build_object_key_index
Build an ObjectKeyIndex for the object at path.
build_structural_index
Build a document-wide Stage-1 structural index (safe).
delete_index
Deletes an element from a JSON array located at the specified path by its index.
delete_key
Deletes a key-value pair from a JSON object located at the specified path.
deny_unknown_keys
Closed-object check: every key on the object at object_path must be in allowed (logical unescaped keys for unescaped on-wire keys).
deny_unknown_keys_at
Like deny_unknown_keys but object_path is a string path ("" or omit root).
escape_json_key
Escape s as it would appear inside a JSON string (no surrounding quotes).
escape_json_string
Serialize s as a JSON string literal (including surrounding quotes).
estimate_projected_len
Ballpark capacity hint for flat keep-lists (not exact projector output).
estimate_values_len
Sum of on-wire value lengths for paths.
find_value
Locates a JSON value byte-slice within a raw JSON buffer by its path.
for_each_jsonl_doc
Map each line as TypedDocRef with a fallible callback.
for_each_jsonl_line
Map JSONL lines with a fallible callback (stream; no intermediate Vec).
from_jshift_bytes
Happy-path alias: bytes → T: JsonView (roadmap adoption surface).
from_json_string
Parse a JSON string literal (including surrounding quotes) into an unescaped String.
insert_array_element
Insert new_element into the array at path so it becomes the element at index.
json_lines
Iterate non-empty lines in a JSONL / NDJSON buffer (zero-copy slices).
jsonl_docs
Zero-copy TypedDocRef per non-empty JSONL line.
jsonl_docs_owned
Owned TypedDoc per line (copies each line into a Vec<u8>).
merge_object_shallow
Shallow-merge patch object members into the object at path.
mutate_value
Mutates a JSON value in-place inside a Vec<u8> buffer by its path.
mutate_value_checked
Like mutate_value, but requires new_value to parse as exactly one complete JSON value (object, array, string, number, true/false/null) with no trailing junk after optional whitespace.
object_from_iter
Convenience: build an object from (key, value) pairs (call order preserved).
parse_jmespath_expr
Parse a JMESPath expression into a SelectExpr.
parse_path
Parses a dot-and-bracket notation path string into zero-copy path segments.
parse_project_path
Parse projection path segments including signed slices and wildcards.
plan_object_fields
Build a ProjectPlan for project_object_fields (array of objects → multi-select hash).
plan_prefers_parallel
Heuristic: prefer Rayon list emit when per-element work is likely CPU-bound.
prepend_to_array
Insert new_element as the first element of the array at path.
project
Project json according to plan, returning a new buffer.
project_as_view
Project json down to T’s keep-list, then decode as T.
project_auto_indexed
Build structural + plan array indexes, then project (one-shot convenience).
project_each
Project each element of a list projection plan as a separate card.
project_each_indexed
Like project_each, using side-tables from doc when the array path is indexed.
project_indexed
Project using a pre-built IndexedDocument.
project_indexed_auto
Project with side-tables, enabling parallel [*] only when plan_prefers_parallel.
project_indexed_prepare
Ensure doc has side-tables for the plan’s array paths, then project.
project_into
Project into an existing buffer (appends).
project_jmespath
Convenience: ProjectPlan::from_jmespath then project.
project_jsonl_write
Write each list-projection card as one NDJSON line (JSON object + newline) to w.
project_jsonl_write_indexed
Indexed NDJSON list projection (reuse side-tables).
project_object_fields
Generic “thin cards” helper: project each object in an array down to named fields.
project_object_fields_each
Thin-card variant: project named fields on each object in array_path, one callback per element (no giant result array).
project_object_fields_each_indexed
Indexed thin-card each-callback (prefer when reusing an IndexedDocument).
project_object_fields_jsonl_write
NDJSON thin cards: each object in array_path → one line with selected fields.
project_parallel_auto
Build plan indexes, then project_indexed_auto (seq vs parallel by heuristic).
project_paths
Convenience: ProjectPlan::from_paths then project.
project_write
Stream projection into any Write without buffering the full output.
projected_len
Exact projected byte length without retaining the output (counting sink).
read_jsonl
Map each non-empty JSONL line through a JsonView reader.
read_jsonl_indexed
Map each line with schema-guided indexing (rebuild index per line).
read_line_indexed
Index a single JSONL line (array paths), then read a view.
read_view
Shared helper: read any JsonView from bytes.
rename_key
Rename an object member key in place (value bytes unchanged).
require_paths
Ensure every path exists (any non-missing value, including JSON null).
require_paths_non_null
Ensure every path exists and is not JSON null.
require_type
Require path to have type expected.
select_from_project_path
Build SelectExpr from a keep-list path.
set_at
Convenience: build from a pre-parsed Path set (avoids re-tokenize in hot loops).
static_array_prefixes_from_path
Extract static array path prefixes from a field path for derive auto-index.
try_parse_path
Strict path parser. Returns Error::InvalidPath for malformed path syntax instead of silently dropping segments.
type_at
Sniff the JSON type at path.
upsert_at
upsert_at_path
Upsert new_value at path, creating missing object parents as {}.
upsert_object_key
Upserts (inserts or updates) a key-value pair in a JSON object located at the specified path.
validate_closed
Closed mode: required paths present and no unknown keys on the root object.
validate_open
Open mode: only check that required paths exist; unknowns are ignored.
write_json_string
Append a JSON string literal (including surrounding quotes) for s into out.
write_json_string_content
Append the escaped form of s (content only, no surrounding quotes) into out.
write_jsonl_docs
Write each document’s bytes as one NDJSON line.
write_jsonl_line
Write one JSON value as a JSONL line (raw + \n).
write_jsonl_views
Write each view as one NDJSON line via TypedDoc::from_view.
write_view
Shared helper: write any JsonView into a buffer.

Derive Macros§

JsonMutatorSchema
Derive typed JSON path readers/mutators and implement [jshift::JsonView].
JsonView
Alias for JsonMutatorSchema: a Rust type is a projection of JSON bytes.