DataCortex

The best standalone JSON/NDJSON compressor. Beats zstd-19 and brotli-11 on every file tested.

DataCortex auto-infers your JSON schema, applies columnar reorg + type-specific encoding, then picks the optimal entropy coder (zstd or brotli). No schema files, no database, no configuration — just datacortex compress data.json.

Benchmarks

Fast mode vs the best general-purpose compressors:

On larger structured logs:

Higher is better. DataCortex wins on every file. Lossless — byte-exact decompression guaranteed.

How it works

Input JSON/NDJSON
  → Format detection (JSON vs NDJSON vs generic)
  → Schema inference (auto-detect column types)
  → Columnar reorg (group values by field)
  → Nested decomposition (flatten objects into sub-columns)
  → Type-specific encoding:
      Integers → delta + ZigZag + LEB128 varint
      Booleans → bitmap (8 per byte)
      Timestamps → epoch micros + delta varint
      Enums → frequency-sorted ordinal dictionary
      Strings → quote strip + length prefix
      UUIDs → 16-byte binary
  → Auto-select best entropy coder (zstd or brotli)
  → .dcx output

The auto-fallback tries 6+ compression paths and picks the absolute smallest output. You always get the best result.

Installation

# From source
git clone https://github.com/rushikeshmore/DataCortex
cd DataCortex
cargo build --release
# Binary at target/release/datacortex

Requires Rust 1.85+ (edition 2024).

Usage

# Compress (auto-detects format, picks best compression)
datacortex compress data.ndjson
datacortex compress api-response.json
datacortex compress logs.ndjson -m fast          # explicit fast mode

# Decompress
datacortex decompress data.dcx output.ndjson

# Benchmark against zstd
datacortex bench corpus/ -m fast --compare

# Higher compression (slower)
datacortex compress data.ndjson -m fast --level 19

# Inspect a .dcx file
datacortex info data.dcx

Compression modes

Fast mode is recommended for JSON/NDJSON. It runs the full preprocessing pipeline (schema inference, columnar reorg, typed encoding) then picks the best entropy coder automatically.

Balanced/Max modes use a bit-level context mixing engine with 13 specialized models. Better for general text but slower.

Why DataCortex beats zstd on JSON

General-purpose compressors (zstd, brotli, gzip) treat JSON as opaque bytes. They find repeated patterns via LZ77 sliding window matching but don't understand the structure.

DataCortex understands JSON:

1. Schema inference — auto-detects that timestamp is a timestamp, status is a low-cardinality enum, user_id is a string
2. Columnar reorg — groups all timestamps together, all status codes together (like Parquet, but automatic)
3. Type-specific encoding — timestamps become tiny delta-encoded varints, booleans become bitmaps, enums become 1-byte ordinals
4. The preprocessed data compresses dramatically better — zstd/brotli on columnar+typed data achieves 2-3x better ratios than on raw JSON

Architecture

datacortex/
  crates/
    datacortex-core/          Core compression library
      src/
        format/               Schema inference, columnar transforms, typed encoding
        model/                CM engine (13 context models)
        mixer/                Triple logistic mixer + 7-stage APM
        entropy/              Binary arithmetic coder
        codec.rs              Pipeline orchestrator + auto-fallback
        dcx.rs                .dcx file format (v3)
    datacortex-cli/           CLI binary
  corpus/                     Test corpus (JSON, NDJSON, text)
  benchmarks/                 Baseline measurements

Development

cargo test                                      # 354 tests
cargo clippy --all-targets -- -D warnings       # lint (0 warnings)
cargo fmt --check                               # formatting
cargo build --release                           # optimized build

License

MIT