JSON Tools RS

A high-performance Rust library for advanced JSON manipulation with SIMD-accelerated parsing, providing unified flattening and unflattening operations through a clean builder pattern API. Ships with Rust, Python, and JVM (Java/Spark) bindings.

Why JSON Tools RS?

JSON Tools RS is designed for developers who need to:

• Transform nested JSON into flat structures for databases, CSV exports, or analytics
• Clean and normalize JSON data from external APIs or user input
• Process large batches of JSON documents efficiently
• Maintain type safety with perfect roundtrip support (flatten → unflatten → original)
• Work with both Rust and Python using the same consistent API

Unlike simple JSON parsers, JSON Tools RS provides a complete toolkit for JSON transformation with production-ready performance and error handling.

Features

• 🚀 Unified API: Single JSONTools entry point for flattening, unflattening, or pass-through transforms (.normal())
• 🔧 Builder Pattern: Fluent, chainable API for easy configuration and method chaining
• ⚡ High Performance: SIMD-accelerated JSON parsing with FxHashMap, SmallVec stack allocation, and tiered caching
• 🚄 Parallel Processing: Built-in Rayon-based parallelism (persistent work-stealing pool) for 3-5x speedup on batch operations and large nested structures
• 🎯 Complete Roundtrip: Flatten JSON and unflatten back to original structure with perfect fidelity
• 🧹 Comprehensive Filtering: Remove empty strings, nulls, empty objects, and empty arrays (works for both flatten and unflatten)
• 🔄 Advanced Replacements: Key/value replacements, literal (exact substring match) by default, or regex by wrapping the pattern in r'...'
• 🛡️ Collision Handling: Intelligent .handle_key_collision(true) to collect colliding values into arrays
• 📅 Date Normalization: Automatic detection and normalization of ISO-8601 dates to UTC
• 🔀 Automatic Type Conversion: Convert strings to numbers, booleans, and nulls with .auto_convert_types(true)
• 📦 Batch Processing: Process single JSON or batches; Python also supports dicts and lists of dicts
• 🐍 Python Bindings: Full Python support with perfect type preservation (input type = output type)
• 📊 DataFrame/Series Support: Native support for Pandas, Polars, PyArrow, and PySpark DataFrames and Series in Python
• ☕ JVM Bindings: Java/Spark UDFs (row and batched mapPartitions tiers) for Databricks Jobs/notebooks on classic compute and other Spark workloads -- see jvm/README.md

Table of Contents

• Why JSON Tools RS?
• Features
• Quick Start
  • Rust Examples
  • Python Examples
  • JVM / Spark Examples
• Quick Reference
• Installation
• Architecture
• Performance
• Contributing
• License
• Changelog

Quick Start

Rust - Unified JSONTools API

The JSONTools struct provides a unified builder pattern API for all JSON manipulation operations. Simply call .flatten() or .unflatten() to set the operation mode, then chain configuration methods and call .execute().

Basic Flattening

use json_tools_rs::{JSONTools, JsonOutput};

let json = r#"{"user": {"name": "John", "profile": {"age": 30, "city": "NYC"}}}"#;
let result = JSONTools::new()
    .flatten()
    .execute(json)?;

if let JsonOutput::Single(flattened) = result {
    println!("{}", flattened);
}
// Output: {"user.name": "John", "user.profile.age": 30, "user.profile.city": "NYC"}

Advanced Flattening with Filtering

use json_tools_rs::{JSONTools, JsonOutput};

let json = r#"{"user": {"name": "John", "details": {"age": null, "city": ""}}}"#;
let result = JSONTools::new()
    .flatten()
    .separator("::")
    .lowercase_keys(true)
    .key_replacement("r'(User|Admin)_'", "")
    .value_replacement("@example.com", "@company.org")
    .remove_empty_strings(true)
    .remove_nulls(true)
    .remove_empty_objects(true)
    .remove_empty_arrays(true)
    .execute(json)?;

if let JsonOutput::Single(flattened) = result {
    println!("{}", flattened);
}
// Output: {"user::name": "John"}

Automatic Type Conversion

Convert string values to numbers, booleans, dates, and null automatically for data cleaning and normalization.

use json_tools_rs::{JSONTools, JsonOutput};

let json = r#"{
    "id": "123",
    "price": "$1,234.56",
    "discount": "15%",
    "active": "yes",
    "verified": "1",
    "created": "2024-01-15T10:30:00+05:00",
    "status": "N/A"
}"#;

let result = JSONTools::new()
    .flatten()
    .auto_convert_types(true)
    .execute(json)?;

if let JsonOutput::Single(flattened) = result {
    println!("{}", flattened);
}
// Output: {
//   "id": 123,
//   "price": 1234.56,
//   "discount": 15.0,
//   "active": true,
//   "verified": 1,
//   "created": "2024-01-15T05:30:00Z", // Normalized to UTC
//   "status": null
// }

Python - Unified JSONTools API

The Python bindings provide the same unified JSONTools API with perfect type matching: input type equals output type.

Basic Usage

import json_tools_rs as jt

# Basic flattening - dict input → dict output
result = jt.JSONTools().flatten().execute({"user": {"name": "John", "age": 30}})
print(result)  # {'user.name': 'John', 'user.age': 30}

# Basic unflattening - dict input → dict output
result = jt.JSONTools().unflatten().execute({"user.name": "John", "user.age": 30})
print(result)  # {'user': {'name': 'John', 'age': 30}}

Advanced Configuration & Parallelism

import json_tools_rs as jt

# Configure tools with parallel processing settings
tools = (jt.JSONTools()
    .flatten()
    .separator("::")
    .lowercase_keys(True)
    .remove_empty_strings(True)
    .parallel_threshold(50)       # Parallelize batches >= 50 items
    .num_threads(4)               # Use 4 threads
    .nested_parallel_threshold(200) # Parallelize large objects
)

# Process a batch of data
batch = [{"data": i} for i in range(100)]
results = tools.execute(batch)

DataFrame & Series Support

import json_tools_rs as jt
import pandas as pd

# Pandas DataFrame input → Pandas DataFrame output
df = pd.DataFrame([
    {"user": {"name": "Alice", "age": 30}},
    {"user": {"name": "Bob", "age": 25}},
])
result = jt.JSONTools().flatten().execute(df)
print(type(result))  # <class 'pandas.core.frame.DataFrame'>

# Also works with Polars, PyArrow Tables, and PySpark DataFrames
# Series input → Series output (Pandas, Polars, PyArrow)

JVM / Spark

JNI-based Java bindings, mirroring the same JSONTools builder, for use as Apache Spark UDFs -- a simple row UDF and a higher-throughput batched mapPartitions transform. Built for Databricks Jobs/notebooks on classic compute and other Spark workloads (not usable inside a Databricks Lakeflow Declarative Pipeline -- Databricks doesn't permit JVM libraries on pipeline compute at all; use the Python bindings above, wrapped in a pandas_udf, for that case instead).

import io.github.amaye15.jsontoolsrs.JsonTools;
import io.github.amaye15.jsontoolsrs.JsonToolsHandle;

try (JsonToolsHandle tools = JsonTools.builder()
        .flatten()
        .separator("::")
        .keyReplacement("r'^admin_'", "")
        .removeNulls(true)
        .build()) {
    String result = tools.execute("{\"admin_name\": \"Jane\", \"age\": null}");
    // {"name":"Jane"}
}

See jvm/README.md for the Spark UDF API and Setting Up on Databricks for the full deployment walkthrough (both this and the pandas_udf path).

Quick Reference

Method Cheat Sheet

Automatic Type Conversion

When .auto_convert_types(true) is enabled, the library performs smart parsing on string values:

1. Date & Time (ISO-8601):

• Detects date strings to avoid converting them to numbers (e.g., "2024-01-01").
• Normalizes datetimes to UTC.
• Supports offsets (+05:00), Z suffix, and naive datetimes.

2. Numbers:

• Basic: "123" → 123, "45.67" → 45.67
• Separators: "1,234.56" (US), "1.234,56" (EU), "1 234.56" (Space)
• Currency: "$123", "€99", "£50", "¥1000", "R$50"
• Scientific: "1e5" → 100000
• Percentages: "50%" → 50.0, "12.5%" → 12.5
• Basis Points: "50bps" → 0.005, "100 bp" → 0.01
• Suffixes: "1K", "2.5M", "5B" (Thousand, Million, Billion)

3. Booleans:

• "true", "false", "yes", "no", "on", "off", "y", "n" (case-insensitive).
• Note: "1" and "0" are treated as numbers, not booleans.

4. Nulls:

• "null", "nil", "none", "N/A" (case-insensitive) → null.

Installation

Rust

cargo add json-tools-rs

Python

pip install json-tools-rs

JVM / Spark

No published artifact yet -- build from source (cargo build --release --features jvm && cd jvm && mvn package), or download the jar from a jvm-ci.yml CI run. See jvm/README.md for details; a Maven Central release (io.github.amaye15:json-tools-rs-spark) ships automatically on tagged releases.

Architecture

The codebase is organized into focused, single-responsibility modules:

src/
├── lib.rs            Facade: mod declarations + pub use re-exports
├── json_parser.rs    Conditional SIMD parser (sonic-rs on 64-bit, simd-json on 32-bit)
├── types.rs          Core types: JsonInput, JsonOutput
├── error.rs          Error types with codes E001-E008
├── config.rs         Configuration structs and operation modes
├── cache.rs          Tiered caching: regex, key deduplication, phf perfect hash
├── convert.rs        Type conversion: numbers, dates, booleans, nulls (SIMD-optimized)
├── transform.rs      Filtering, key/value replacements, collision handling
├── flatten.rs        Flattening algorithm with Rayon parallelism
├── unflatten.rs      Unflattening with SIMD separator detection
├── builder.rs        Public JSONTools builder API and execute() entry point
├── python.rs         Python bindings via PyO3
├── jvm.rs            JVM bindings via JNI (Java/Spark UDFs, see jvm/)
├── tests.rs          Unit tests
└── main.rs           CLI examples

The processing pipeline:

1. Parse -- SIMD-accelerated JSON parsing (json_parser)
2. Flatten/Unflatten -- Recursive traversal with Arc<str> key dedup (flatten/unflatten)
3. Transform -- Lowercase, replacements (cached regex), collision handling (transform)
4. Filter -- Remove empty strings, nulls, empty objects/arrays (transform)
5. Convert -- Type conversion with first-byte discriminators (convert)
6. Serialize -- Output to JSON string or native Python types

Performance

Benchmark Results

Measured on Apple Silicon. Results may vary by platform and data shape.

Optimization Techniques

JSON Tools RS uses several techniques to achieve high performance (~2,000+ ops/ms):

• SIMD-JSON: Hardware-accelerated parsing via sonic-rs (64-bit) / simd-json (32-bit).
• SIMD Byte Search: memchr/memmem for SIMD-accelerated string operations and pattern matching.
• FxHashMap: Faster hashing for string keys with rustc-hash.
• Tiered Caching: Three-level regex cache (compile-time phf table → thread-local FxHashMap → global RwLock<FxHashMap>).
• SmallVec & Cow: Stack allocation for depth stacks and number buffers; zero-copy string handling.
• Arc<str> Deduplication: Shared key storage to minimize allocations in wide/deep JSON.
• First-Byte Discriminators: Rapid rejection of non-convertible strings during type conversion.
• Parallelism: Rayon's persistent work-stealing thread pool for batch processing and large nested structures (avoids per-call OS thread spawn cost).

CLI Demo

The crate includes an educational demo binary that showcases library features:

cargo run

This prints progressive examples covering basic flattening, unflattening, custom separators, filtering, replacements, collision handling, type conversion, and batch processing.

Contributing

See CONTRIBUTING.md for development setup, testing, benchmarking, and PR guidelines.

Changelog

v0.9.2 (Current)

(v0.9.1 was tagged a day earlier but only completed publishing to Maven Central -- a crates.io/PyPI release pipeline bug caused those two to fail before any upload. Fixed and re-cut as v0.9.2 across all three registries; no code changes beyond the release pipeline fix itself.)

• JVM (Java) bindings (BREAKING for key_replacement/value_replacement, see below): new Spark UDF bindings (jvm/) with full feature parity, via a JNI shim over the same Rust core -- see jvm/README.md.
• key_replacement/value_replacement pattern syntax (BREAKING): patterns are now literal (exact substring match) by default; wrap in r'...' (e.g. r'^admin_') for regex. Previously every pattern was always compiled as regex.
• Rayon parallelism: batch processing switched back from std::thread::scope (per-call OS thread spawn) to Rayon's persistent work-stealing pool -- measurably faster for small-to-medium batches.
• has_escape scanner bug fix: escape sequences not adjacent to a quote (\n, \t, \r, \uXXXX) were previously invisible to the tape scanner, silently skipping auto_convert_types/replacements/lowercase_keys for affected strings.
• crates.io and Maven Central publishing enabled on tagged releases.

See CHANGELOG.md for full details on all of the above.

v0.9.0

• Crossbeam Parallelism: Migrated from Rayon to Crossbeam for finer-grained parallel control.
• DataFrame/Series Support: Native Python support for Pandas, Polars, PyArrow, and PySpark DataFrames and Series.
• Modular Architecture: Refactored into 10 focused modules for maintainability (zero API changes).
• Performance Optimizations: Eliminated per-entry HashMap in parallel flatten, early-exit discriminators, SIMD literal fallback, thread-local regex cache half-eviction, vectorized clean_number_string().
• Python Binding Optimizations: mem::take for zero-cost builder mutations, O(1) DataFrame/Series reconstruction.

v0.8.0

• Python Feature Parity: Added auto_convert_types, parallel_threshold, num_threads, and nested_parallel_threshold to Python bindings.
• Enhanced Type Conversion: Added support for ISO-8601 dates, currency codes (USD, EUR), basis points (bps), and suffixed numbers (K/M/B).
• Date Normalization: Automatic detection and UTC normalization of date strings.

See CHANGELOG.md for full history.