jshift
jshift is a schema-guided, 100% safe Rust JSON path reader and in-place mutator.
It is built for high-performance middleboxes, API gateways, webhook routers, edge filters, and data pipelines that need to selectively query and modify JSON on raw byte buffers—without the full AST allocate / serialize cycle of traditional parsers.
If you only need “read one field,” other path engines exist.
If you need read + rewrite the same buffer under real production constraints (safe Rust, no second serialize pass), jshift is the product.
Why jshift exists
Typical stack today:
bytes → serde_json::from_slice → Value tree → change a field → to_vec → bytes
That is correct and ergonomic. It is also expensive when:
- documents are large (MBs of payload, one hot field),
- you process millions of JSONL lines,
- you run many concurrent workers on the same shape of traffic,
- you only care about one path (or a small schema of paths).
jshift flips the model:
bytes → path scan to byte offsets → splice / shift bytes in place → same Vec<u8>
No tree. No second full document serialize. Reads return zero-copy slices into the original buffer.
Real-world example: API JSON ingestion (what IT teams do today)
A common pattern in platform / integration teams looks like this:
- Pull a large JSON payload from a partner or SaaS API (catalog, events, orders, product dump—often multi‑MB or hundreds of MB).
- Ingest into an internal service: validate a couple of fields, stamp metadata (
ingested_at,source,tenant_id), maybe drop or rewrite a status flag, then forward the body to a queue, object store, or downstream microservice. - Today’s default stack is almost always:
HTTP body → serde_json::from_slice(or equivalent) → walk a full in-memory tree → change one or two fields →to_vec/ re-serialize → publish. That is simple to write and easy to reason about—but on a 300 MB catalog you still allocate and walk the entire tree, then allocate and write another ~300 MB of output, even if you only neededproducts[0].titleand a top-levelstatus. Under bursty multi-worker ingestion, that becomes CPU, memory, and GC (or allocator) pressure, not “JSON is slow because HTTP is slow.” - With jshift, the same job is: keep the body as
Vec<u8>→ path-scan only the fields you need → splice stamps or flags in place with safe byte rotations → hand the same buffer downstream. Peers that only need a header field never pay for the giantproductsarray. Teams that must stay on safe Rust (nounsafehot loops) get selective R/W without becoming a second full parser.
Impact in practice: lower p99 on hot ingestion paths, less memory headroom for concurrent workers, and fewer “we only touch three fields but clone the whole document” incidents—while serde remains the right tool when you truly need a full typed domain model.
Key Features
- Path-selective scans: Walk only the path you need on raw
&[u8]/Vec<u8>. - In-place mutations: Upsize and downsize with safe slice rotations (compiles to
memmove-class moves). - 100% Safe Rust:
#![forbid(unsafe_code)]—noget_uncheckedin the hot path. - Zero-copy reads:
find_valuereturns a subslice of the input buffer. - Schema derive:
#[derive(JsonMutatorSchema)]generates typed readers and mutators with compile-time path constants. - Object & array CRUD: Update, upsert, delete keys; append, index, delete elements; nested
upsert_at_path. - Correct string encoding:
ToJsonBytesand key upserts escape",\, and control characters. - Owned + pointer paths:
Path,try_parse_path, JSON Pointer (Path::from_json_pointer). - Option / null: first-class for training JSONL and partial records.
Contract: jshift is a non-validating path engine. It assumes mostly well-formed JSON along the path you traverse. Callers must supply complete JSON value bytes for raw mutations (or use
ToJsonBytes/mutate_value_checked).
When to use jshift vs serde_json
| You should use… | When… |
|---|---|
| jshift | You touch few fields on large or high-volume JSON; you want in-place updates; you control or trust path shape; latency / throughput matter. |
| serde_json | You need a full typed model, validation of the whole document, arbitrary transforms, or you already pay for a complete parse. |
| Both | Parse selectively with jshift for hot paths; use serde when a request actually needs the full document. |
Rule of thumb
- “Filter / tag / rewrite
statuson every JSONL line” → jshift. - “Deserialize into
struct Request { … }with dozens of fields and nested enums” → serde. - “Gateway: inspect
headers.x-request-id, maybe setstatus, forward body” → jshift.
How it works under the hood (byte shifts, not magic)
You do not need a systems-programming background to use jshift—but understanding the mechanism explains the speed.
1. Find = locate byte offsets, not build a tree
find_value(json, path) walks the raw bytes along a path like user.score or tags[0]:
- Skip whitespace and nested structures you do not need (including large arrays/objects on the way).
- Match object keys (as on-wire bytes between quotes) or array indexes.
- Return
(start, end)into the original buffer—or just the slicejson[start..end].
So for:
a path "target" means: skip the entire data array as one bulk value, then read the number.
No heap tree of every object in data.
2. Mutate = splice into the same Vec<u8>
When you replace a value:
| Case | What jshift does |
|---|---|
Same length (e.g. 123456 → 999999) |
Overwrite bytes in place. |
| Longer | Vec::resize, then tail.rotate_right(delta) to open a gap, write the new value. |
| Shorter | tail.rotate_left(delta) to close the gap, then truncate. |
rotate_left / rotate_right on a slice are safe APIs. LLVM typically lowers them to the same class of bulk memory moves as memmove. You get high performance without writing unsafe pointer arithmetic.
Deletes work the same way: compute a span (including commas), expand over adjacent whitespace for tidy output (“pretty delete”), then shift the tail left.
3. Why that is faster than serde for selective work
serde’s happy path is:
- Parse every token into a
Value(or a typed struct). - Mutate the tree.
- Walk the tree and write a new document.
jshift’s happy path is:
- Scan until the path of interest.
- Move only the tail after the edit.
If the document is 10MB and you change six bytes near the end, serde still rebuilds ~10MB of output. jshift moves a tail and rewrites the field.
4. Safe vs “unchecked” path engines (gjson)
gjson is an excellent, battle-tested read engine. Its hottest skip loops use techniques like continuous bulk scans—and in the Rust port, unchecked indexing (get_unchecked) in the inner loop. That can buy speed on “skip a huge array to a trailing key.”
jshift deliberately does not.
| Choice | jshift | Typical max-speed finder (e.g. gjson hot path) |
|---|---|---|
unsafe |
Forbidden (forbid(unsafe_code)) |
Used in hot loops for fewer bounds checks |
| Primary goal | Find and mutate with byte offsets | Find / query |
| Failure mode on bad input | Result / syntax errors along the path |
Often best-effort on malformed JSON |
| Dependency / trust surface | One crate, auditable safe code | Speed via unsafe assumptions |
We did steal the ideas that transfer cleanly to safe Rust:
- bulk “squash” of nested containers,
- tight string skipping,
- unrolled scans over non-structural bytes,
…and left the remaining gap on pure “key-last + 10MB array” finds as a conscious trade for memory safety and mutator correctness. If your only requirement is the absolute fastest read and you accept unsafe, evaluate gjson. If you need safe in-place mutation, jshift is the fit.
Performance (illustrative — with marketing multipliers)
Fresh Criterion run on this repo’s benches/json_benchmark.rs (quiet machine, release, ~6 s measurement windows). Means below are the criterion mid estimate. Re-run before putting numbers on a slide deck—absolute times vary by CPU, but ratios are the story.
Legend
- jshift — path scan / in-place mutate,
#![forbid(unsafe_code)] - gjson — path read (hot skip path may use unchecked loads)
- sonic-rs — high-performance JSON library (pointer get)
- serde_json — full
Valueparse (+ re-serialize on mutate)
Find: path engines + full parse
| Workload | jshift | gjson | sonic-rs | serde_json | jshift vs serde | jshift vs gjson | jshift vs sonic |
|---|---|---|---|---|---|---|---|
| Key-last ~10MB (target after huge array) | ~11.0 ms | ~4.9 ms | ~27.4 ms | ~223 ms | ~20× faster | ~2.3× slower | ~2.5× faster |
| Key-first ~10MB (target is first field) | ~39 ns | ~87 ns | ~97 ns | ~222 ms | ~5,700,000× faster | ~2.2× faster | ~2.5× faster |
| Small ~1KB top-level key | ~39 ns | ~89 ns | ~96 ns | ~6.1 µs | ~160× faster | ~2.3× faster | ~2.5× faster |
Small ~1KB nested meta.ver |
~80 ns | ~143 ns | ~153 ns | ~6.4 µs | ~80× faster | ~1.8× faster | ~1.9× faster |
One-liner: On selective finds, jshift is typically ~20×–millions× faster than full serde_json parse, and on most shapes here ~2× faster than gjson/sonic—while remaining 100% safe Rust; gjson can still win pure “skip a giant trailing array” finds (~2×).
Yes, the key-first row looks absurd—that is the point. Serde still parses the entire multi‑megabyte document. jshift matches the first key and stops.
Path-engine honesty
- On key-last + giant array, gjson can win pure find (~2× here). We optimize that path in safe code, still beat sonic-rs (~2.5×), and still crush full parse (~20× vs serde).
- On key-first and small / nested finds in this run, jshift leads gjson and sonic-rs while remaining safe.
- jshift’s killer feature is not “always #1 find”; it is find → mutate same buffer.
Mutate: the product story
| Workload | jshift | serde_json (parse + set + to_vec) |
jshift vs serde |
|---|---|---|---|
| Key-last ~10MB (same-length overwrite) | ~11.3 ms | ~198 ms | ~18× faster |
| Small ~1KB | ~76 ns | ~7.3 µs | ~95× faster |
One-liner: Selective in-place mutate is where jshift shines—about ~18× (large) to ~100× (small) versus “parse whole tree, change one field, re-serialize.”
This is the “gateway / JSONL cleaner / feature-flag rewrite” workload: change a field, keep shipping the rest of the bytes.
Concurrent find — 8 independent workers
Same model for every engine: eight workers, each extracts target from a shared buffer. No shared parse tree. That means serde re-parses eight times.
Key-last ~10MB (must skip the array)
| Engine | Mean (8-way wall) | vs serde | vs gjson |
|---|---|---|---|
| gjson ×8 | ~10.3 ms | ~43× faster | 1× |
| jshift ×8 | ~20.1 ms | ~22× faster | ~2.0× slower |
| serde_json ×8 | ~442 ms | 1× | — |
One-liner: Under 8-way key-last load, jshift is still ~22× faster than parallel full parses; gjson leads pure read (~2×) on this shape.
Key-first ~10MB (early exit)
| Engine | Mean (8-way wall) | vs serde | vs gjson |
|---|---|---|---|
| jshift ×8 | ~18.7 µs | ~22,000× faster | ~1.02× (≈tie / slight win) |
| gjson ×8 | ~19.1 µs | ~21,000× faster | 1× |
| serde_json ×8 | ~404 ms | 1× | — |
One-liner: When the hot field is near the front—the common API case—eight workers finish in ~19 µs with jshift vs ~400 ms of full re-parses (~22,000×).
# Full suite
# Head-to-head find (jshift / gjson / sonic-rs / serde_json)
# Parallel groups (key-last + key-first)
Large optional real-world fixtures (100MB+) belong outside crates.io and preferably outside this git history—see benches/README.md.
These numbers are not a claim that jshift is always fastest for every JSON task. For full typed deserialization and schema validation of entire documents, use serde.
Installation
[]
= "0.2"
Requires a recent stable Rust (see rust-version in Cargo.toml; edition 2024).
Quick Start: AI dataset cleaning (JSONL)
In pipelines that filter or tag training samples, records are often JSON Lines. Instead of parse → allocate → re-serialize millions of lines, update fields in-place on raw bytes.
1. Define a schema
use JsonMutatorSchema;
The derive emits 'static path constants (no re-parsing the path string on every set_*).
2. Selective reads
let mut line = b"{\"instruction\": \"Translate...\", \"tokens\": 1024, \"status\": \"pending\", \"tags\": [\"llm\"]}".to_vec;
let record = read_from_json.unwrap;
assert_eq!;
assert_eq!;
3. In-place mutations & array append
let mut mutator = mutator;
if record.tokens > 512
let updated = read_from_json.unwrap;
assert_eq!;
assert_eq!;
Low-level API
use ;
let mut json = b"{\"user\": \"farmer\", \"score\": 9.5}".to_vec;
let path = parse_path;
// Or reuse an owned path:
// let path = Path::parse("score");
assert_eq!;
mutate_value.unwrap;
// Prefer checked mutate when the payload might be garbage:
mutate_value_checked.unwrap;
// Strings are escaped for you:
mutate_value.unwrap;
// Create nested parents if missing:
let mut empty = b"{}".to_vec;
upsert_at_path.unwrap;
Optional fields (null / missing)
How mutations work (detail)
When a replacement changes size, jshift resizes the Vec<u8> and rotates the tail:
Upsize: [prefix | OLD | tail....]
resize
[prefix | OLD | ....tail | pad]
rotate_right(delta)
[prefix | gap | tail....]
write NEW
[prefix | NEW | tail....]
Downsize: [prefix | OLDVALUE | tail]
rotate_left(delta)
[prefix | NEW | tail | garbage]
truncate
[prefix | NEW | tail]
Deletes:
- Locate the member (keys use logical names; matching uses escaped on-wire form).
- Expand the delete span to include the correct comma and adjacent whitespace (pretty delete →
{}/[]when empty). - Shift the tail left.
Capabilities
Object operations
mutate_value/mutate_value_checked— overwrite at a pathupsert_object_key— insert or update a key (logical keys, JSON-escaped on write)upsert_at_path— upsert a leaf and create missing object parents as{}delete_key— remove a key/value, fix commas, trim whitespace
Array operations
append_to_array— append with comma injectionarray_len— count elements without allocatingdelete_index— remove an element, fix commas, pretty-trim
Paths
- Dot / bracket:
metadata.tags[0].nameviaparse_path/try_parse_path/Path - JSON Pointer:
Path::from_json_pointer("/a~1b/0") - Derive validates paths at compile time and caches segment tables
Convert
FromJsonSlice/ToJsonBytesfor numbers, bool,String,Vec,Optionfrom_json_string/escape_json_string/escape_json_key
Related crates
| Crate | Role | Prefer when… |
|---|---|---|
| serde_json | Full AST / typed serde | Complete models, validation, general transforms |
| gjson | Fast path reads (may use unsafe in hot loops) | Read-only queries; absolute max skip speed |
| sonic-rs / simd-json | High-performance parse | Full document parse with speed focus |
| jshift | Path scan + in-place mutate + schema derive + safe | Selective R/W, JSONL, gateways, safe-only codebases |
Development
See CONTRIBUTING.md and CHANGELOG.md.
License
Licensed under either of:
- Apache License, Version 2.0 (LICENSE-APACHE)
- MIT license (LICENSE-MIT)
at your option.