DataCortex

The best standalone JSON/NDJSON compressor. Beats zstd-19 and brotli-11 on every file tested.

Site | crates.io | PyPI | Docs

DataCortex auto-infers your JSON schema, applies columnar reorg + type-specific encoding, then picks the optimal entropy coder (zstd or brotli). No schema files, no database, no configuration. Just datacortex compress data.json.

Benchmarks

Fast mode vs the best general-purpose compressors:

On larger structured logs:

Higher is better. DataCortex wins on every file. Lossless, byte-exact decompression guaranteed.

Performance

Throughput on an Apple M-series chip (Fast mode, single run, release build):

Decode is near-instant (176-1124 MB/s). Encode trades speed for 2x better compression vs zstd. For throughput-critical pipelines, DataCortex is best suited as a batch compressor for log storage, not a real-time codec.

Run datacortex bench corpus/ -m fast --compare to measure on your hardware.

Installation

Rust:

cargo install datacortex-cli

Python:

pip install datacortex

From source:

git clone https://github.com/rushikeshmore/DataCortex
cd DataCortex
cargo build --release

Requires Rust 1.85+.

Usage

# Compress (auto-detects format, picks best compression)
datacortex compress data.ndjson
datacortex compress api-response.json
datacortex compress logs.ndjson -m fast          # explicit fast mode

# Decompress
datacortex decompress data.dcx output.ndjson

# Streaming (pipe-friendly)
cat logs.ndjson | datacortex compress - -o compressed.dcx
datacortex decompress compressed.dcx -o -        # stdout

# Chunked compression (for large NDJSON)
datacortex compress logs.ndjson -o out.dcx --chunk-rows 10000

# Custom dictionary (for known schemas)
datacortex train-dict corpus/*.ndjson --output my.dict
datacortex compress logs.ndjson --dict my.dict

# Benchmark against zstd
datacortex bench corpus/ -m fast --compare

# Higher compression (slower)
datacortex compress data.ndjson -m fast --level 19

# Inspect a .dcx file
datacortex info data.dcx

Compression modes

Fast mode is recommended for JSON/NDJSON. It runs the full preprocessing pipeline (schema inference, columnar reorg, typed encoding) then picks the best entropy coder automatically.

Balanced/Max modes use a bit-level context mixing engine with 13 specialized models. Better for general text but slower.

Python

import datacortex

compressed = datacortex.compress(json_bytes, mode="fast")
original = datacortex.decompress(compressed)

# File-based
datacortex.compress_file("logs.ndjson", "logs.dcx", mode="fast")
datacortex.decompress_file("logs.dcx", "logs.json")

# Format detection
fmt = datacortex.detect_format(data)  # "ndjson", "json", "generic"

How it works

1. Format detection - auto-identifies JSON, NDJSON, or generic data
2. Schema inference - discovers column types (integer, boolean, timestamp, enum, string, etc.)
3. Columnar reorg - transposes row-oriented NDJSON into column-oriented layout
4. Type-specific encoding - delta+varint for integers, bitmaps for booleans, epoch deltas for timestamps
5. Auto-fallback - tries 6+ compression paths (zstd, brotli, with/without preprocessing) and picks the smallest

No schema files. No configuration. Fully automatic.

Development

cargo test                                      # 390 tests
cargo clippy --all-targets -- -D warnings       # lint (0 warnings)
cargo fmt --check                               # formatting
cargo build --release                           # optimized build

License

MIT