Skip to main content

Crate jshift

Crate jshift 

Source
Expand description

jshift — Schema-guided, safe in-place JSON path reader and mutator.

This crate provides a 100% safe Rust engine to selectively read, mutate, upsert, and delete values inside raw JSON byte buffers (&[u8] and Vec<u8>) without building a full AST. Path scans return zero-copy slices; mutations resize the buffer and shift the tail with safe slice rotations.

§What jshift is (and is not)

DoDon’t
Path index + mutateFull JSON DOM as the eval spine
Schema-guided projection (JsonView) / TypedDocMulti-format serde (bincode, …)
Safe structural side-tablesPromise simdjson Stage-1 crowns
Preserve unmentioned fieldsFull RFC validator (unless you add one)

Open documents: paths you don’t name are left unread (and byte-preserved on write). That is a feature for API evolution (same spirit as prost unknown fields).

§Cargo features

FeatureDefaultPurpose
deriveyesJsonMutatorSchema / JsonView (structs + externally tagged enums)
serdenoAsJshiftView Deserialize/Serialize bridge — see the serde bridge guide
axumnoJshiftJson / JshiftDoc extractors
parallelnoRayon-backed parallel list projections on large indexed arrays
dhat-heapnodhat global allocator for examples/alloc_profile (local profiling only)

Core path engine and structural indexing always compile. Indexing is opt-in at the call site (IndexedDocument, read_from_indexed); never taxed on default finds. Parallel project is opt-in (project_parallel / project_indexed_parallel); use project_indexed_auto / project_parallel_auto to enable Rayon only when plan_prefers_parallel says the plan is CPU-heavy. Streaming cards: project_each / project_jsonl_write (no giant output array).

jshift = { version = "0.7", features = ["derive"] }
# core only (no proc-macros):
jshift = { version = "0.7", default-features = false }
# axum extractors / serde bridge:
jshift = { version = "0.7", features = ["axum", "serde"] }
# bulk array project with rayon:
jshift = { version = "0.7", features = ["parallel"] }

Measurement: Criterion (benches/), RSS (scripts/measure_rss.sh), dhat/heaptrack (examples/alloc_profile). Large fixtures: scripts/fetch_teefury.sh + scripts/build_large_catalog.sh → gitignored benches/data/large.json.

CI verifies both default and --no-default-features builds.

§Features

§Quick Start

use jshift::{find_value, mutate_value, parse_path};

let mut json = b"{\"user\": \"farmer\", \"score\": 9.5}".to_vec();
let path = parse_path("score");

// Read value
let score_bytes = find_value(&json, &path).unwrap();
assert_eq!(score_bytes, b"9.5");

// Mutate in-place
mutate_value(&mut json, &path, b"10.0").unwrap();
assert_eq!(json, b"{\"user\": \"farmer\", \"score\": 10.0}".to_vec());

§TypedDoc: typed fields without Value

use jshift::{JsonDoc, TypedDoc};

let mut doc = TypedDoc::from_slice(br#"{"status":"ok","id":7,"items":[{"n":1}],"extra":true}"#);
assert_eq!(doc.get::<u64>("id").unwrap(), 7);
assert_eq!(doc.get_str("status").unwrap(), "ok");
// sparse array field stream (relative path parsed once)
assert_eq!(doc.collect_each_get::<u64>("items", "n").unwrap(), vec![1]);

doc.mutate().set("status", "accepted").unwrap();
assert_eq!(doc.get_str("status").unwrap(), "accepted");
assert!(doc.contains("extra").unwrap());

§JsonView: typed projections (open partial structs)

use jshift::{JsonView, JsonMutatorSchema};

#[derive(JsonMutatorSchema)]
struct ListingCard {
    #[json(path = "id")]
    id: u64,
    #[json(path = "title")]
    title: String,
    // intentionally no variants / images: partial view
}

fn ingest<T: JsonView>(buf: &[u8]) -> Result<T, jshift::Error> {
    T::read_from(buf)
}

let json = br#"{"id":7,"title":"Hat","images":[1,2,3],"variants":[]}"#;
let card: ListingCard = ingest(json).unwrap();
assert_eq!(card.id, 7);
assert_eq!(card.title, "Hat");

// write_into only touches named paths; other keys stay put
let mut buf = json.to_vec();
card.write_into(&mut buf).unwrap();
assert!(buf.windows(8).any(|w| w == b"\"images\""));

§Enums (external default; also internal / adjacent / untagged)

use jshift::JsonView;

#[derive(Debug, PartialEq, JsonView)]
enum Status {
    Ready,
    #[json(rename = "in_progress")]
    Running,
    Done,
}

#[derive(Debug, PartialEq, JsonView)]
#[json(tag = "type")]
enum InternalEv {
    Ready,
    Click {
        #[json(path = "x")]
        x: u64,
        #[json(path = "y")]
        y: u64,
    },
}

assert_eq!(Status::read_from(br#""Ready""#).unwrap(), Status::Ready);
assert_eq!(
    InternalEv::read_from(br#"{"type":"Click","x":1,"y":2}"#).unwrap(),
    InternalEv::Click { x: 1, y: 2 }
);

Also: #[json(tag = "t", content = "c")] (adjacent), #[json(untagged)]. See enums and api_guide.

§Byte flatten (0.7 / Track B)

use jshift::JsonView;

#[derive(Debug, PartialEq, JsonView)]
struct Meta {
    #[json(path = "created_at")]
    created_at: String,
    #[json(path = "source")]
    source: String,
}

#[derive(Debug, PartialEq, JsonView)]
struct Event {
    #[json(path = "id")]
    id: u64,
    #[json(flatten)]
    meta: Meta,
}

let raw = br#"{"id":7,"created_at":"t0","source":"api","noise":true}"#;
let e = Event::read_from(raw).unwrap();
assert_eq!(e.meta.source, "api");
let out = e.to_schema_bytes().unwrap();
assert!(!out.windows(6).any(|w| w == b"\"meta\"")); // flat wire

§Project: keep-list / JMESPath subset → smaller JSON

use jshift::{project_jmespath, project_paths, ProjectPlan, ProjectStyle};

let json = br#"{"id":7,"title":"Hat","images":[1,2,3],"meta":{"x":1}}"#;
let out = project_paths(json, &["id", "title", "meta.x"]).unwrap();
assert_eq!(out, br#"{"id":7,"title":"Hat","meta":{"x":1}}"#);

// Keep-list wildcards preserve ancestor keys:
let catalog = br#"{"products":[{"sku":"A","blob":1},{"sku":"B","blob":2}]}"#;
assert_eq!(
    project_paths(catalog, &["products[].sku"]).unwrap(),
    br#"{"products":[{"sku":"A"},{"sku":"B"}]}"#
);

// JMESPath subset: multi-select hash + rename (result is the projected array):
assert_eq!(
    project_jmespath(catalog, "products[*].{sku: sku}").unwrap(),
    br#"[{"sku":"A"},{"sku":"B"}]"#
);

let pretty = ProjectPlan::from_paths(&["id"]).unwrap().style(ProjectStyle::Pretty { indent: 2 });
let _ = jshift::project(json, &pretty).unwrap();

§High-Impact Real-World Use Case: LLM Dataset Processing (JSONL)

In AI training pipelines (e.g., LoRA finetuning), datasets are stored as JSONL files. You can inspect token lengths and mark records as skipped or cleaned in-place:

use jshift::JsonMutatorSchema;

#[derive(JsonMutatorSchema)]
struct TrainingRecord {
    #[json(path = "tokens")]
    tokens: usize,
    #[json(path = "status")]
    status: String,
}

let mut line = b"{\"instruction\": \"Translate...\", \"tokens\": 1024, \"status\": \"pending\"}".to_vec();

// Parse selectively
let record = TrainingRecord::read_from_json(&line).unwrap();

// Skip long contexts in-place!
if record.tokens > 512 {
    let mut mutator = TrainingRecord::mutator(&mut line);
    mutator.set_status("skipped").unwrap();
}

assert_eq!(
    line,
    b"{\"instruction\": \"Translate...\", \"tokens\": 1024, \"status\": \"skipped\"}".to_vec()
);

Re-exports§

pub use enums::is_external_object_shape;
pub use enums::is_external_unit_shape;
pub use enums::payload_as_view;
pub use enums::peek_external_object;
pub use enums::read_adjacent;
pub use enums::read_external;
pub use enums::read_external_object;
pub use enums::read_external_unit;
pub use enums::read_internal;
pub use enums::read_tag_field;
pub use enums::read_untagged;
pub use enums::write_adjacent_content;
pub use enums::write_adjacent_unit;
pub use enums::write_external_object;
pub use enums::write_external_unit;
pub use enums::write_internal_fields;
pub use enums::write_internal_unit;
pub use enums::ExternalTag;

Modules§

api_guide
Product map: which API for which job (ingest, views, project, enums, bridges).
enums
Enum tagging free functions and types (also re-exported at crate root). Enum tagging on raw JSON (serde-compatible shapes, no Value tree).
serde_bridge_guide
Serde bridge guide

Structs§

ArrayBuilder
Fluent builder for a JSON array ([...]).
ArrayElems
Fallible iterator over JSON array element spans.
ArrayIndex
Side-table for one array value: start offset of each element.
AsJshiftView
Newtype: serde ↔ JsonView via raw JSON text (not a Value tree).
AsJshiftViewOwned
Like AsJshiftView, but deserializes via Box<RawValue> (owned buffer).
BatchPlan
Fluent collector for MutateOps, then commit onto a buffer.
CountingSink
Counts bytes that would be written (exact projected length without retaining output).
HashField
One field in a multi-select hash (output_key: expr).
IndexedDocument
A JSON buffer plus structural / array / object indexes for path-accelerated reads.
IndexedMapView
MapView after MapView::index — O(1) lookup for unescaped keys.
IndexedValueList
ValueList after ValueList::index — O(1) element access.
IndexedViewList
ViewList after ViewList::index — O(1) element access by index.
JshiftDoc
Axum extractor/response for a raw TypedDoc (owned body bytes).
JshiftJson
Axum extractor/response for T: JsonView.
JsonLines
Iterator over non-empty JSONL lines as byte slices.
JsonWriter
Imperative JSON encoder into one growable buffer.
JsonlDecoder
Incremental JSONL decoder for chunked input (HTTP body, growing file, pipe).
Limits
Limits applied by check_document / check_depth.
MapView
Cursor over a JSON object whose values decode as T: FromJsonSlice.
NestedView
Borrowed JSON subtree treated as a document root for relative access.
ObjectBuilder
Fluent builder for a JSON object ({...}).
ObjectEntries
Fallible iterator over object key/value spans (DynObject cursor).
ObjectEntry
One object member: on-wire key content + raw value span.
ObjectKeyIndex
Map of object keys (on-wire bytes between quotes) → value span.
ObjectKeyReport
Result of scanning one object’s members under a win policy.
ObjectSelect
Object field selection for subset projection.
Path
Owned, reusable JSON path.
ProjectPlan
Compiled projection plan: selection AST + formatting + missing policy.
RawJson
Owned raw JSON value bytes (object, array, string, number, bool, or null).
ResolvedObjectKey
One logical key after applying First/Last (or unique Error success).
SharedDocument
Cheaply cloneable, read-only JSON buffer.
StreamPos
Where a record came from in a multi-source stream.
StructuralIndex
Sorted absolute offsets of structural characters outside JSON strings: { } [ ] : ,
TransformPipeline
Ordered list of Transform steps.
TypedDoc
Owned JSON buffer with typed path access and exclusive in-place mutation.
TypedDocRef
Borrowed, read-only typed document over &[u8].
TypedMutator
Exclusive mutator for a TypedDoc.
ValueList
Cursor over a JSON array of values decoded via FromJsonSlice.
ValueListIter
Iterator yielding Result<T, Error> for each primitive/array element.
ViewList
Cursor over a JSON array of objects decoded as T: JsonView.
ViewListIter
Iterator yielding Result<T, Error> for each array element.
WriteSink
Streams projected bytes to W (no full-output buffer).

Enums§

ArraySelect
Array projection strategy.
CmpOp
Comparison operators (JMESPath).
CollectPolicy
How to materialize a ViewList (or similar cursor).
Collected
Result of ViewList::collect under a CollectPolicy.
DuplicateKeyPolicy
How to treat repeated keys in a JSON object.
Error
Errors returned by scanning and mutating operations.
JshiftRejection
Rejection when jshift JSON extraction fails.
JsonType
Type-check a path as a JSON string / number / bool / object / array without full decode.
MissingPolicy
Behavior when a selected path is missing in the input.
MutateOp
One mutation in a batch plan.
OwnedPathSegment
Owned path segment (see Path).
PathSegment
Represents a segment in a JSON path (either a string key or a zero-indexed array index).
Presence
Whether a path is absent, JSON null, or a non-null value.
ProjectPathSegment
One step of a keep-list path (plus wildcards / slices).
ProjectStyle
How projected structure is formatted.
RootKind
Kind of a JSON value at the document root (or any single-value span).
SelectExpr
Selection / expression node over a JSON value span.
Transform
One transform step in a pipeline.

Traits§

FromJsonSlice
Trait implemented by types that can be parsed directly from a raw JSON byte slice.
JsonDoc
Shared typed read surface over any JSON byte buffer.
JsonView
A Rust type that is a projection of JSON bytes along named paths.
ToJsonBytes
Trait implemented by types that can be serialized directly into a JSON byte representation.

Functions§

append_object_members_raw
Append all members of a JSON object into an open object buffer mid-build.
append_to_array
Appends a new value to the end of a JSON array located at the specified path.
apply_ops
Apply ops in order to a raw buffer.
array_from_iter
Convenience: build an array from values.
array_len
Returns the number of elements in the array located at the specified path.
build_array
Build a root array by driving a JsonWriter callback.
build_array_index
Build an ArrayIndex for the array at path.
build_object
Build a root object by driving a JsonWriter callback.
build_object_key_index
Build an ObjectKeyIndex for the object at path.
build_schema_bytes
Build schema-only bytes from any JsonView (roadmap build-from-schema).
build_structural_index
Build a document-wide Stage-1 structural index (safe).
check_depth
Measure max nesting depth of the root value; error if > max_depth.
check_document
Reject oversized buffers and excessive nesting (single structural pass).
check_duplicate_keys
Scan the root object: resolve wins (First/Last) or reject (Error).
check_duplicate_keys_at
Scan the object at path (empty = root).
check_duplicate_keys_deep
Alias: deep scan with DuplicateKeyPolicy::Error only.
check_duplicate_keys_path
Path string helper ("" = root).
delete_index
Deletes an element from a JSON array located at the specified path by its index.
delete_key
Deletes a key-value pair from a JSON object located at the specified path.
deny_unknown_keys
Closed-object check: every key on the object at object_path must be in allowed (logical unescaped keys for unescaped on-wire keys).
deny_unknown_keys_at
Like deny_unknown_keys but object_path is a string path ("" or omit root).
escape_json_key
Escape s as it would appear inside a JSON string (no surrounding quotes).
escape_json_string
Serialize s as a JSON string literal (including surrounding quotes).
estimate_projected_len
Ballpark capacity hint for flat keep-lists (not exact projector output).
estimate_values_len
Sum of on-wire value lengths for paths.
find_value
Locates a JSON value byte-slice within a raw JSON buffer by its path.
for_each_jsonl_doc
Map each line as TypedDocRef with a fallible callback.
for_each_jsonl_line
Map JSONL lines with a fallible callback (stream; no intermediate Vec).
for_each_jsonl_multi
Stream every non-empty JSONL line across many buffers (files/shards) in order.
for_each_jsonl_view_multi
Like for_each_jsonl_multi, decoding each line as T: JsonView.
for_each_page_array
Each page is a root JSON array; stream each element as T: JsonView.
for_each_page_doc
Each page/file is one JSON document (object or array root).
for_each_page_item_raw
Stream raw array element spans from the array at items_path across pages.
for_each_page_items
Each page is a JSON object; stream T from the array at items_path.
from_jshift_bytes
Happy-path alias: bytes → T: JsonView (roadmap adoption surface).
from_json_string
Parse a JSON string literal (including surrounding quotes) into an unescaped String.
from_raw_value
Decode T from a RawValue (works for borrowed or boxed).
from_slice_view
Decode T from JSON bytes (direct jshift; no serde).
from_str_view
Decode T from a JSON text string (direct jshift; no serde intermediate).
from_value
Boundary: serde_json::Value → T: JsonView (text intermediate).
insert_array_element
Insert new_element into the array at path so it becomes the element at index.
json_lines
Iterate non-empty lines in a JSONL / NDJSON buffer (zero-copy slices).
jsonl_docs
Zero-copy TypedDocRef per non-empty JSONL line.
jsonl_docs_owned
Owned TypedDoc per line (copies each line into a Vec<u8>).
measure_depth
Report peak nesting depth without failing (still errors on syntax).
merge_field_paths
Collect keep-list paths for project plans: own paths + each flattened child’s FIELD_PATHS (derive convention).
merge_flatten_object
Spread every member of child_object into parent (root object) by shallow upsert of each key.
merge_object_shallow
Shallow-merge patch object members into the object at path.
merge_patch
Apply RFC 7396 merge patch to the document root.
merge_patch_at
Apply merge patch to the value at path (empty path = root).
mutate_value
Mutates a JSON value in-place inside a Vec<u8> buffer by its path.
mutate_value_checked
Like mutate_value, but requires new_value to parse as exactly one complete JSON value (object, array, string, number, true/false/null) with no trailing junk after optional whitespace.
object_from_iter
Convenience: build an object from (key, value) pairs (call order preserved).
object_key_wins
Resolve which value span wins for one key_raw (on-wire content between quotes).
object_member_count
Count members in an object (for tests / diagnostics).
parse_jmespath_expr
Parse a JMESPath expression into a SelectExpr.
parse_path
Parses a dot-and-bracket notation path string into zero-copy path segments.
parse_project_path
Parse projection path segments including signed slices and wildcards.
plan_object_fields
Build a ProjectPlan for project_object_fields (array of objects → multi-select hash).
plan_prefers_parallel
Heuristic: prefer Rayon list emit when per-element work is likely CPU-bound.
prepend_to_array
Insert new_element as the first element of the array at path.
project
Project json according to plan, returning a new buffer.
project_as_view
Project json down to T’s keep-list, then decode as T.
project_auto_indexed
Build structural + plan array indexes, then project (one-shot convenience).
project_each
Project each element of a list projection plan as a separate card.
project_each_indexed
Like project_each, using side-tables from doc when the array path is indexed.
project_indexed
Project using a pre-built IndexedDocument.
project_indexed_auto
Project with side-tables, enabling parallel [*] only when plan_prefers_parallel.
project_indexed_parallel
Like project_indexed, but enables parallel array [*] emission when a side-table is present and the array is large enough (feature parallel).
project_indexed_prepare
Ensure doc has side-tables for the plan’s array paths, then project.
project_into
Project into an existing buffer (appends).
project_jmespath
Convenience: ProjectPlan::from_jmespath then project.
project_jsonl_write
Write each list-projection card as one NDJSON line (JSON object + newline) to w.
project_jsonl_write_indexed
Indexed NDJSON list projection (reuse side-tables).
project_object_fields
Generic “thin cards” helper: project each object in an array down to named fields.
project_object_fields_each
Thin-card variant: project named fields on each object in array_path, one callback per element (no giant result array).
project_object_fields_each_indexed
Indexed thin-card each-callback (prefer when reusing an IndexedDocument).
project_object_fields_jsonl_write
NDJSON thin cards: each object in array_path → one line with selected fields.
project_object_fields_parallel
Parallel variant of project_object_fields (feature parallel).
project_parallel
Parallel list projection using Rayon (requires feature parallel).
project_parallel_auto
Build plan indexes, then project_indexed_auto (seq vs parallel by heuristic).
project_paths
Convenience: ProjectPlan::from_paths then project.
project_plan_with_flatten
Build a crate::ProjectPlan from own paths plus flattened children’s FIELD_PATHS slices.
project_view_collect
Collect array elements as Vec<T: JsonView> (explicit owned collect).
project_view_each
Stream each element of the array at array_path as T: JsonView.
project_write
Stream projection into any Write without buffering the full output.
projected_len
Exact projected byte length without retaining the output (counting sink).
read_btreemap_at
Read BTreeMap<String, T> from the JSON object at path.
read_flattened
Decode T from the parent object buffer (paths relative to that root).
read_flattened_opt
Decode Option<T>: None when a required path is missing (PathNotFound).
read_hashmap_at
Read HashMap<String, T> from the JSON object at path.
read_jsonl
Map each non-empty JSONL line through a JsonView reader.
read_jsonl_indexed
Map each line with schema-guided indexing (rebuild index per line).
read_jsonl_multi
Iterator over (StreamPos, line) across multiple JSONL buffers.
read_line_indexed
Index a single JSONL line (array paths), then read a view.
read_view
Shared helper: read any JsonView from bytes.
read_view_at
Read nested T: JsonView at a path (child value becomes the view’s root).
read_view_at_opt
Optional nested view: missing path or JSON null → None.
rename_key
Rename an object member key in place (value bytes unchanged).
require_paths
Ensure every path exists (any non-missing value, including JSON null).
require_paths_non_null
Ensure every path exists and is not JSON null.
require_type
Require path to have type expected.
select_from_project_path
Build SelectExpr from a keep-list path.
set_at
Convenience: build from a pre-parsed Path set (avoids re-tokenize in hot loops).
static_array_prefixes_from_path
Extract static array path prefixes from a field path for derive auto-index.
to_value
Boundary: T: JsonView → serde_json::Value via schema bytes.
try_parse_path
Strict path parser. Returns Error::InvalidPath for malformed path syntax instead of silently dropping segments.
type_at
Sniff the JSON type at path.
upsert_at
upsert_at_path
Upsert new_value at path, creating missing object parents as {}.
upsert_object_key
Upserts (inserts or updates) a key-value pair in a JSON object located at the specified path.
validate_closed
Closed mode: required paths present and no unknown keys on the root object.
validate_open
Open mode: only check that required paths exist; unknowns are ignored.
validate_unique_keys
Security shorthand: reject any duplicate key at the root object.
validate_unique_keys_at
Reject duplicates on the object at path.
validate_unique_keys_deep
Recursively reject duplicate keys in every nested object (unique-only, no reports).
validate_unique_keys_path
Reject duplicates on the object at path string ("" = root).
write_btreemap_at
Replace the object at path with a JSON encoding of map (sorted keys).
write_flattened
Open-upsert a flattened child into parent (preserves unknown parent keys).
write_flattened_opt
Open-upsert when Some; no-op when None (does not delete child keys).
write_hashmap_at
Replace the object at path with a JSON encoding of map.
write_json_string
Append a JSON string literal (including surrounding quotes) for s into out.
write_json_string_content
Append the escaped form of s (content only, no surrounding quotes) into out.
write_jsonl_docs
Write each document’s bytes as one NDJSON line.
write_jsonl_line
Write one JSON value as a JSONL line (raw + \n).
write_jsonl_views
Write each view as one NDJSON line via TypedDoc::from_view.
write_map_iter_at
Encode a string-key map as a JSON object and replace the value at path.
write_view
Shared helper: write any JsonView into a buffer.
write_view_at
Open nested write: apply view.write_into on the child value at path.
write_view_at_closed
Closed nested write: replace the value at path with view.to_schema_bytes().
write_view_at_closed_opt
Closed nested write when Some; no-op when None.
write_view_at_open
Open nested write (same as write_view_at; explicit name for call sites).
write_view_at_opt
Open nested write when Some; no-op when None (does not delete the path).

Derive Macros§

JsonMutatorSchema
Derive typed JSON path readers/mutators and implement [jshift::JsonView].
JsonView
Alias for JsonMutatorSchema: a Rust type is a projection of JSON bytes.