NextJson

English Documentation - 中文文档

Wiki

The repository Wiki is published from the /wiki directory: GitHub Wiki

NextJson is a data-contract engine for Rust: a dependency-free, no_std + alloc library built for controlled protocols and resource-constrained environments. It is not "another serde" and it does not claim to replace Postcard for device-to-device links. What it does is make three properties first-class:

1. schema-first — a type does not only encode and decode, it describes the contract. Every derived type carries const SCHEMA: TypeSchema, a compile-time metadata tree that can be introspected at runtime, rendered as JSON Schema, used to validate incoming data, and diffed against a previous release to detect protocol breakage.
2. multi-format — switching wire formats for the same type is a first-class operation, not an adapter. Twenty-one formats share one NsonSerialize / NsonDeserialize implementation through the format-neutral FormatEncoder / FormatDecoder contracts, and format-to-format relay through the shared event stream is verified to be byte-identical to direct encoding.
3. reuse-first — sustained decoding prioritizes memory and allocation reuse. Typed decode streams straight into your fields through checked DecodeSlot state (no intermediate tree, no placeholder value), unescaped strings borrow the input buffer, and the unified token stream lets content be replayed without taxing the hot path.

For high-frequency device-to-device communication, what matters is byte count, determinism, version compatibility and latency — a unified API does not win those by itself. NextJson's job is the contract layer around the wire: describe it, validate what enters, and detect when a change breaks the other side.

Guarantees

• The workspace contains only the local nextjson and nextjson-derive crates.
• The only dependency entry is the local workspace derive crate. There are no registry, Git, or external path dependencies.
• The derive crate uses only Rust's standard proc_macro API.
• The core enables no_std, denies unsafe code, and denies missing docs.
• Native contracts are named nextencode, nextdecode, and nextdecode_into.
• Unescaped JSON strings and definite CBOR text strings can borrow input bytes.
• JSON/CBOR conversion relays events without constructing an intermediate tree.

Audit the complete build graph directly:

cargo tree --workspace --all-features --edges normal,build,dev

The output must contain only the two local packages:

nextjson
└── nextjson-derive (local workspace proc-macro)
nextjson-derive

Installation

Default std + derive configuration:

[dependencies]

nextjson = "0.1"

Core no_std + alloc configuration:

[dependencies]

nextjson = { version = "0.1", default-features = false }

Enable the repository-owned derive macros without enabling std:

[dependencies]

nextjson = { version = "0.1", default-features = false, features = ["derive"] }

Feature	Default	Purpose
std	yes	Standard I/O adapters and standard-library-specific types
derive	yes	Repository-owned NsonSerialize and NsonDeserialize derives
simd	no	Opt-in architecture acceleration for JSON string scanning: SSE2 + runtime-detected AVX2 on x86-64, NEON on aarch64, with a portable register-width fallback elsewhere. The unsafe code is confined to the scan module and gated behind this feature; default builds remain #![deny(unsafe_code)] with zero unsafe.

Native API

nextencode(&value) returns compact JSON bytes. nextdecode(input) decodes one complete JSON value and rejects trailing data. Writer, reader, pretty-print, dynamic Value, JSON macro, schema inspection, and JSON Schema APIs remain available as focused helpers.

The design is data-model-first (not AST-first): typed decode streams directly into your fields with zero intermediate tree, and Value is an opt-in consumer of the same decoder. Two encoder policies are exposed: Encoder validates the event protocol on every call, while FastEncoder (the nextencode / to_vec / to_string / writer entry points) trusts the derive-verified call sequence and skips per-value checks for ~2x encoding throughput.

use nextjson::{NsonDeserialize, NsonSerialize};

#[derive(Debug, PartialEq, NsonSerialize, NsonDeserialize)]
#[njson(rename_all = "camelCase")]
struct User {
    user_id: u64,
    name: String,
    #[njson(default)]
    tags: Vec<String>,
}

let expected = User {
    user_id: 7,
    name: "Ada".into(),
    tags: vec!["compiler".into()],
};

let bytes = nextjson::nextencode(&expected)?;
let actual: User = nextjson::nextdecode(&bytes)?;
assert_eq!(actual, expected);
# Ok::<(), nextjson::Error>(())

The contract pillars

Pillar 1 — schema as a compile-time contract

Every derived type exposes const SCHEMA: TypeSchema, a Copy metadata tree built in const context. It is not documentation that can drift: it is generated from the same attribute parse that drives encode and decode, so the description and the wire behavior cannot disagree.

# use nextjson::{NsonDeserialize, NsonSerialize};
#[derive(NsonSerialize, NsonDeserialize)]
struct Point { x: i32, y: i32 }

let schema = nextjson::schema_of::<Point>();
let json_schema = nextjson::to_json_schema::<Point>(); // draft-07 style
# let _ = (schema, json_schema);

The schema tree is the input to the two other pillars: validation and version-compatibility checking.

Pillar 2 — safety policy as part of the schema

A schema can declare what the type allows to enter the system, not just what it looks like. Limits are declared with derive attributes and carried inside SCHEMA; a validator (nextjson::validate) walks a decoded Value against the schema and reports every violation, plus the paths of sensitive values for redaction.

use nextjson::{NsonDeserialize, NsonSerialize};

#[derive(NsonSerialize, NsonDeserialize)]
#[njson(max_depth = 4, deny_unknown_fields)]
struct Request {
    #[njson(max_str_len = 64)]
    name: String,
    #[njson(max_items = 100, min = 0, max = 1000)]
    samples: Vec<i32>,
    #[njson(sensitive)]
    token: String,
}

let input = br#"{"name":"NextJson","samples":[1,2,3],"token":"secret"}"#;
let decoded: nextjson::Value = nextjson::nextdecode(input)?;
let report = nextjson::validate_value::<Request>(&decoded);
assert!(report.is_ok());          // policy passed
for v in &report.violations {     // or inspect every violation
    // e.g. ViolationKind::StringTooLong { max: 64 } at path "name"
}
// report.sensitive == ["token"] — redact before logging

Declared limits (all optional, all const-constructible):

Attribute	Scope	Enforced on
max_str_len = N	field / newtype variant	string length in Unicode scalar values
max_items = N	field / newtype variant	array elements / object entries
min = N / max = N	field / newtype variant	numbers (inclusive, exact for i128/u128)
sensitive	field / newtype variant	reported, never rejected (redaction)
max_depth = N	container	container nesting below this type
deny_unknown_fields	container	unknown keys on structs and tagged enums

Runtime tuning goes through ValidateConfig: a global nesting cap (max_depth) and a message-size bound (max_message_size + the actual message_len), which is a byte-layer concern the value walker cannot measure itself.

Validation is a post-decode gate: it runs on an already-materialized Value and never touches the hot decode path. It collects all violations in one pass (fail-collect, not fail-fast) so a production gate can log every offending path at once.

Pillar 3 — version compatibility as a schema diff

Because the schema is a value, protocol evolution becomes a pure function: nextjson::check(old_schema, new_schema) reports every change that can break an old reader consuming new data (forward) or a new reader consuming old data (backward), with per-issue severity.

use nextjson::{check_between, Severity, NsonDeserialize, NsonSerialize};

#[derive(NsonSerialize, NsonDeserialize)] struct V1 { id: u64, name: String }
#[derive(NsonSerialize, NsonDeserialize)] struct V2 { id: u64, name: String, email: String }

let report = check_between::<V1, V2>();
assert!(!report.backward_compatible); // old data lacks the new required field
assert!(report.forward_compatible);
assert_eq!(report.worst_severity(), Some(Severity::Critical));

Detected classes:

Change	Severity	Direction affected
Added required field	Critical	backward
Removed required field	Critical	forward
Field / variant renamed	Critical	both
Type-family change (string→number, struct→seq, ...)	Critical	both
Added / removed enum variant	Critical	forward / backward
Tag representation change (tag / content / untagged)	Critical	both
Optional field became required	Critical	backward
Float became integer	Critical	backward
Integer range narrowed	Warning	backward
Integer became float	Warning	forward
Required field became optional	Warning	forward
Default value changed	Note	— (semantic)
Safety policy changed	Note	— (not wire-breaking)

This is a static report: it cannot know the actual values in the field. A Warning (e.g. i32 → u8) is safe only if the real data never exceeds the new range. Run it in CI on every release candidate.

Cross-format architecture

cross_format::EventSink is NextJson's own dependency-free structural protocol. json_into and cbor_into are sources; JsonSink and CborSink are destinations. Both directions validate event order and nesting. The built-in CBOR implementation supports an RFC 8949 JSON-compatible profile, including 128-bit bignums and finite IEEE floats.

use nextjson::cross_format;

let json = br#"{"name":"NextJson","values":[1,2,3],"ok":true}"#;
let cbor = cross_format::json_to_cbor(json)?;
let json_again = cross_format::cbor_to_json(&cbor)?;

let left: nextjson::Value = nextjson::nextdecode(json)?;
let right: nextjson::Value = nextjson::nextdecode(&json_again)?;
assert_eq!(left, right);
# Ok::<(), nextjson::Error>(())

API	Purpose
json_into	Relay JSON input into any repository-owned EventSink
cbor_into	Relay CBOR input into any repository-owned EventSink
json_to_cbor / json_to_cbor_writer	Stream JSON into CBOR
cbor_to_json / cbor_to_json_writer	Stream CBOR into JSON
cbor_to_json_pretty	Stream CBOR into formatted JSON

The CBOR profile accepts definite and indefinite arrays, maps, and text; u64/i64 major types; tag 2/tag 3 bignums for exact u128/i128 values; and finite half-, single-, and double-precision floats. Map keys must be UTF-8 text.

The profile intentionally rejects values that JSON cannot preserve: arbitrary byte strings, non-text map keys, non-finite floats, and unknown semantic tags. No lossy fallback is performed.

Multi-format engine

nextjson::formats is a dependency-free, format-neutral codec engine. The crate's own NsonSerialize / NsonDeserialize contracts are generic over FormatEncoder / FormatDecoder, so one implementation serves every format whose wire model can represent that value. Most encoders emit directly; document-shaped TOML and YAML collect a Value first so tables can be ordered correctly. Unsupported combinations return errors listed in the matrix below.

Event-order validation is centralized: the format encoders and the cross-format sinks drive one shared protocol state machine, parameterized only by whether the wire format has explicit array separators (JSON does, CBOR does not). On the decode side, the byte lexer serves typed scalar reads directly from the source byte, so the unified token stream stays available for content replay without taxing the hot path.

use nextjson::formats;

let value = ("NextJson", vec![1_u64, 2, 3], true);

let json = formats::encode_with(&value, formats::Json)?;
let msgpack = formats::encode_with(&value, formats::MsgPack)?;
let yaml = formats::encode_with(&value, formats::Yaml)?;

let back: (String, Vec<u64>, bool) = formats::decode_with(&json, formats::Json)?;
assert_eq!(back, formats::decode_with(&msgpack, formats::MsgPack)?);
assert_eq!(back, formats::decode_with(&yaml, formats::Yaml)?);
# Ok::<(), nextjson::Error>(())

Twenty-one formats are registered. Formats are first-class Format values with a canonical name, MIME type, file extensions, and binary/text classification, so they can be passed around, stored, or selected dynamically:

use nextjson::formats::{FormatKind, self};

let kind: Option<FormatKind> = formats::by_extension("toml");
let detected: Option<FormatKind> = formats::detect(br#"{"a":1}"#);
let json = formats::encode_with(&42_i64, formats::Json)?; // format by value
# let _ = (kind, detected, json);

Group	Formats
Text, self-descr.	json, json5, hjson, yaml, toml, ron, sexpr, csv, urlform, ndjson, ini, edn
Binary, self-descr.	cbor, msgpack, ubjson, smile, bson, bencode, pickle
Binary, schema-light	postcard
Environment	envy (deserialization only, requires std)

Transcoding between compatible format models needs no typed value:

use nextjson::formats;
let json = br#"{"name":"NextJson","values":[1,2,3]}"#;
let msgpack = formats::transcode(json, formats::Json, formats::MsgPack)?;
let json2 = formats::transcode(&msgpack, formats::MsgPack, formats::Json)?;
assert_eq!(json2, json);
# Ok::<(), nextjson::Error>(())

Capability matrix (honest limits)

Every format implements the unified contract. Wire-model limits and deliberate codec-subset limits are reported as errors instead of silent lossy fallback:

Format	Scalar Types	Container Types	Features and Limitations
JSON	null, bool, int, float, str	array, object	RFC 8259，完整模型
JSON5	同 JSON + Infinity / NaN	array, object（+ 注释、未加引号键、单引号、尾随逗号）	编码器输出严格 JSON
Hjson	同 JSON	array, object（+ 未加引号键/字符串、注释）	编码器输出严格 JSON
YAML	null, bool, int, float, str	块式 + 流式子集	块式 map/序列（key: value、-、---、{...}/[...]）；块标量 `
TOML	bool, int, float, str（无 null）	表、数组、内联表、多行字符串	拒绝裸标量根；支持 """/''' 多行字符串与 \ 续行；支持 10/16/8/2 进制整数（含 _ 分隔符）；严格校验日期时间形态（TOML 1.0 四种形态：offset/local date-time, date, time）后保留为字符串
RON	bool, int, float, str, char	map, seq, 元组, 结构体, 枚举	Some(...) 包装可双向往返
S-expr	原子、带引号字符串、数字、#t/#f, nil	列表（map 编为 alist）	无模式 Value 解码嵌套 map 存在歧义，请使用类型化目标
CSV	int, float, bool, str	行、带表头的对象行	RFC 4180
Urlform	int, float, bool, str	仅扁平 key/value map	RFC 3986 百分号编码
CBOR	null, bool, int, float, str	array, map	RFC 8949 JSON 兼容 Profile；原生定长容器编解码（兼容读取不定长）；128 位整数走 bignum 标签 2/3；拒绝字节串、非文本键、非有限浮点与未知标签
MessagePack	nil, bool, int, float, str	array, map	JSON 兼容标量/容器族；不支持 bin/ext；拒绝超出 64 位的 128 位整数；非有限浮点线上无损透传，但中继到无法表示它们的格式（JSON、CBOR）时报错
UBJSON	null, bool, int, float, str	array, object	UBJSON v5/Draft 12；对象键为 <整数长度><UTF-8>（无 S 标记）；整数级联最小类型（i/U/I/l/L），超 63 位走 H 高精度十进制；字节串写为 [ $U #n ]；解码兼容计数式 #、强类型 $ 容器（无结束标记）；拒绝非有限浮点
SMILE	null, bool, int, float, str	array, object	Jackson Smile 1.0（0x3A 0x29 0x0A 头）；zigzag VInt 整数、7 位打包浮点、tiny/short/long ASCII/Unicode 字符串；字节串 0xFD 原始二进制；解码兼容共享字符串/键名引用；编码禁用共享（自包含）；拒绝非有限浮点
NDJSON	同 JSON	每行一个 JSON 值	编码：顶层数组逐元素一行；解码：Vec<T> 为行流（跳过空行，容忍 \r），单值模式解析首行并拒绝尾随
INI	str（数字/布尔按文本）	全局段 + [section] 块	注释 ;/#、可选引号（' 字面 / " 转义）；重复键取后者；拒绝数组、null 与嵌套段
EDN	nil, bool, int, float, str	向量 [...]、列表 (...)（→ 数组）、映射 {...}	Clojure EDN 子集；字符串或关键字键（关键字解码为其名）；支持 #_ 丢弃；拒绝符号、集合、字符、tagged literal 与高精度 M/N 数字
BSON	null, bool, int32, int64, double, str	document, array	文档形态（拒绝裸标量根）
Bencode	整数, UTF-8 字符串	list, dict	Key 规范排序；无 null/float；bool 映射为 1/0
Postcard	null, bool, 无符号整数, str	seq, map	非自描述：拒绝有符号整数、float、Option、Value 和 peek
Pickle	None, bool, int, float, str	list, dict, tuple	CPython 协议 2 子集；128 位整数经 LONG1 处理
Envy	int, float, bool, str	扁平 map（环境变量）	仅反序列化；需要 std

detect() is heuristic and intentionally conservative: it claims only strong structural signatures (pickle protocol header, bencode intro, BSON length prefix, text-format ASCII starts, MessagePack/CBOR binary signatures, the SMILE :)\n header, and UBJSON {S… / [$ / [# object/typed-array starts) and returns None for ambiguous input.

Cross-language compatibility

The codecs are verified with explicit foreign-wire fixtures, not only self-round-trips: MessagePack bytes matching Python msgpack, CBOR bytes matching Python cbor2, CPython 3 protocol-2 pickle bytes, canonical bencode, MongoDB-style BSON documents, and hand-written TOML/YAML/RON/S-expression/ JSON5/Hjson inputs. See the formats integration tests for the exact bytes.

Format-equivalence verification

The claim "one data model, many wire formats, no lossy fallback" is verified as an automated equivalence matrix in tests/equivalence.rs:

• Transcode is byte-identical to direct encoding — for every pair of the JSON-compatible family (JSON, JSON5, Hjson, YAML, RON, CBOR, MessagePack), relaying a value from one format into another through the event stream produces exactly the bytes the destination encoder would produce directly.
• Randomized differential — a deterministic LCG generates 200 nested values; each is relayed across the whole family and must stay byte-identical.
• Boundary values — exact i128/u128, f64 extremes, -0.0, Unicode scalar boundaries and control characters travel through every wire format that can represent them.
• Ambiguity semantics — duplicate keys resolve to the last occurrence everywhere; unknown fields are preserved by the schema-less Value consumer.

This platform has caught real codec bugs (JSON5/Hjson \u escapes and surrogate pairs, YAML single-quoted scalars folding line breaks, integral floats losing their float-ness in text codecs, YAML root empty containers emitting no bytes) and keeps them from regressing.

Zero-copy scope

Zero-copy applies when source bytes are already the target UTF-8 string: unescaped JSON strings and definite CBOR text. Escaped JSON strings and indefinite CBOR text require materialization. Output encoding necessarily writes new bytes to its destination. These boundaries are tested with pointer-range assertions.

Derives and schemas

The repository-owned derives support structs, tuple structs, generics, const generics, and external, internal, adjacent, or untagged enum representations. Container attributes include rename_all (including the directional serialize/deserialize form), tag, content, untagged, deny_unknown_fields, default, transparent, crate, bound (including directional bound(serialize=…, deserialize=…)), into, from, try_from, remote, and expecting (overrides the type description used in deserialization error messages; derived implementations install it on the decoder, so container-level type mismatches like begin_object hitting [ report the type name instead of a bare '{'; the default is the type's fully qualified path). Field attributes include rename, alias, default, skip, directional skips, skip_serializing_if, flatten, borrow, with, serialize_with, deserialize_with, getter, and the safety-policy attributes max_str_len, max_items, min, max, sensitive. Variant attributes include rename, rename_all, skip, directional skips, and (on newtype variants) the same safety-policy attributes, which apply to the contained field. Attributes are accepted in #[njson(...)], #[nextjson(...)], or #[serde(...)] form, so existing serde types migrate without rewriting their attributes.

This is not a Serde drop-in guarantee. Visitor/error semantics and external adapters (notably big integers, fixed bytes, curve points, and feature-gated types) require separate verification; see the Serde compatibility contract.

The derive macro is implemented entirely with the standard proc_macro API (no syn, quote, or proc-macro2). The trade-off is stated plainly: a hand-written parser cannot offer the same span-precise diagnostics as a full syn port, so the macro fails loudly with a named message when it does not understand an item (including any future Rust syntax it has not seen), rather than emitting impls from a mis-parsed subset. Generics, where clauses, lifetimes, paths, PhantomData, and all four enum representations are supported and covered by integration tests.

Every derived type also exposes a const SCHEMA: TypeSchema:

# use nextjson::{NsonDeserialize, NsonSerialize};
#[derive(NsonSerialize, NsonDeserialize)]
struct Point { x: i32, y: i32 }

let schema = nextjson::schema_of::<Point>();
let json_schema = nextjson::to_json_schema::<Point>();
# let _ = (schema, json_schema);

Safety and limits

The library contains no unsafe Rust. Checked decode slots prevent an invalid custom implementation from exposing uninitialized memory. Nesting is bounded; numeric conversions are checked; malformed UTF-8, syntax, trailing input, and unrepresentable cross-format values are errors. Applications must still impose deployment-specific byte, collection, time, and output limits.

from_slice / from_str operate on a complete in-memory input; from_reader (std) pulls incrementally from any std::io::Read source (see StreamDecoder). The default JSON and CBOR nesting limit is 128. See the Safety Model for the auditable invariants and remaining application responsibilities.

Examples

Six complete, runnable programs live in nextjson/examples/ (each returns Result and prints its results; run with cargo run -p nextjson --example <name>):

Example	Demonstrates
contract_engine	schema-first: #[njson] policy attributes compiled into SCHEMA, a validation gate over hostile payloads, JSON Schema export, and version-compatibility check_between
multi_format	one value through all 14 wire formats: encoded size, exact round-trip, cross-format transcode chain
cross_format_relay	streaming JSON ⇄ CBOR relay with no intermediate Value, plus writer variants and batch size comparison
zero_copy_reuse	borrowed &str / Bytes decode (pointer-verified against the input slice) and DecodeSlot reuse in a sustained decode loop
streaming_reader	incremental decode from any std::io::Read source (from_reader, StreamDecoder) over a chunked "slow socket" reader
custom_codec	hand-written NsonSchema / NsonSerialize / NsonDeserialize and #[njson(with = "module")] field codecs

cargo run -p nextjson --example contract_engine

Benchmark

The repository-owned benchmark compares encode/decode throughput and encoded size across the 19 wire formats that can represent the fixture (of the 21 registered: envy reads the process environment rather than a wire format, and urlform only represents a flat map). It imports no comparison library in the workspace and does not claim universal superiority.

cargo bench --locked -p nextjson --bench format_comparison

An out-of-workspace crate (benchmarks/serde-comparison/) additionally benchmarks the same data against eleven serde-ecosystem formats: JSON (serde_json and simd-json), JSON5 (serde_json5), YAML (serde_yaml), RON (ron), MessagePack (rmp-serde), CBOR (ciborium), TOML (toml), BSON (bson), postcard (postcard), and bincode (bincode; nextjson has no bincode codec, so that case is labelled na). It also measures a string-heavy long-text JSON fixture, which is the workload where the simd feature's accelerated string scanning matters most. The Vec<Record> fixture covers signed / float / nested values; document-shaped or unsigned-only formats (TOML, BSON, postcard) use a Config fixture. Every format runs a round-trip self-check before measurement. The crate keeps its own Cargo.lock so the workspace dependency audit stays intact.

cd benchmarks/serde-comparison && cargo run --release

Output is CSV (case,size_bytes,encode_ops,encode_MBps,decode_ops,decode_MBps); the per-case window is tuned with NEXTJSON_BENCH_MS (default 2000 ms). The GitHub Actions workflow (.github/workflows/benchmark.yml) runs both suites with the simd feature enabled and merges them into benchmarks/results/Github_Action_Benchmark.md, uploaded as a workflow artifact and committed back on main / manual dispatch / the weekly schedule. See the Reproducible Benchmark for the fixture, measurement method, output format, and reporting requirements.

Reproducibility

Use the committed lock file and run:

cargo fmt --all -- --check
cargo test --workspace --all-features --locked
cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
cargo check -p nextjson --no-default-features --locked
cargo check -p nextjson --all-features --locked   # then verify MSRV with the pinned toolchain
cargo doc --workspace --all-features --no-deps --locked
cargo tree --workspace --all-features --edges normal,build,dev

The lock file must contain only the two local packages. Benchmark reports must include CPU, OS, Rust version, measurement duration, and every output row. Results from one fixture or machine are not evidence of universal performance.

License

Apache-2.0