rusty_zstd 0.1.0

A ground-up, pure-Rust Zstandard (RFC 8878) compressor and decompressor. Levels -7..22 with all nine libzstd strategies, dictionaries + trainer, long-distance matching, seekable frames, multi-threading. Interoperable both directions with facebook/zstd v1.5.7, dual-gated per commit. Zero dependencies, no C, no *-sys, no FFI; builds on no_std + alloc and wasm32. MIT OR Apache-2.0.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142

# rusty_zstd

[![crates.io](https://img.shields.io/crates/v/rusty_zstd?logo=rust)](https://crates.io/crates/rusty_zstd)
[![docs.rs](https://img.shields.io/docsrs/rusty_zstd?logo=docsdotrs)](https://docs.rs/rusty_zstd)
[![CI](https://github.com/Remade-With-Rust/rusty_zstd/actions/workflows/ci.yml/badge.svg)](https://github.com/Remade-With-Rust/rusty_zstd/actions/workflows/ci.yml)
[![License: MIT OR Apache-2.0](https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-blue)](https://github.com/Remade-With-Rust/rusty_zstd#license)
[![Remade With Rust](https://img.shields.io/badge/Remade%20With-Rust-000?logo=rust&logoColor=fff)](https://github.com/Remade-With-Rust)
[![By Mata Network](https://img.shields.io/badge/by-Mata%20Network-5b2be0)](https://www.mata.network/)

> **A ground-up, pure-Rust [Zstandard](https://facebook.github.io/zstd/)
> ([RFC 8878](https://datatracker.ietf.org/doc/html/rfc8878)) compressor and
> decompressor.** `#![deny(unsafe_code)]` everywhere but one audited SIMD island,
> **zero dependencies**, no C, no `*-sys` crate, no FFI. Every frame it emits
> decompresses in facebook/zstd v1.5.7 and every frame that emits decompresses
> here — dual-gated on the Silesia corpus every commit.

Part of **[Remade With Rust](https://github.com/Remade-With-Rust)** by
**[Mata Network](https://www.mata.network/)**.
Full README, benchmark boards and methodology:
**[the repository](https://github.com/Remade-With-Rust/rusty_zstd)**.

---

## Install

```sh
cargo add rusty_zstd
```

```toml
[dependencies]
rusty_zstd = "0.1"

# …or for embedded / wasm targets with no `std`:
rusty_zstd = { version = "0.1", default-features = false, features = ["alloc"] }
```

The minimum supported configuration is `no_std + alloc` — every entry point
returns or fills a `Vec`, so `alloc` is required. MSRV is **1.85**.

| Feature | Default | What it adds |
|---|:--:|---|
| `std` | ✅ | `std::io`-shaped streaming, multi-threading, the trainer, runtime ISA dispatch |
| `alloc` | ✅ | implied by `std`; the minimum supported configuration |
| `profile` | | the in-process stage profiler and its counters (off = zero overhead) |

## Quick start

```rust
use rusty_zstd::{compress, decompress, DEFAULT_CLEVEL};

let data = b"the quick brown fox jumps over the lazy dog ".repeat(64);

let packed = compress(&data, DEFAULT_CLEVEL)?;   // level 3, as libzstd defaults
assert!(packed.len() < data.len());
assert_eq!(decompress(&packed)?, data);          // lossless, always
# Ok::<(), rusty_zstd::Error>(())
```

Levels run **−7…22** with all nine libzstd strategies behind them.
`compress_with` takes a `CompressOptions` for the checksum flag, content size and
dictionary ID; `compress_with_advanced` exposes the full `AdvancedOptions` —
window log, strategy, LDM, workers, job size, overlap.

Streaming, for data that does not fit in memory. The pump is libzstd-shaped: hand
it an input slice and an output buffer, and it reports what it consumed and
produced.

```rust
use rusty_zstd::{compress_stream_out_size, Compressor, Flush};

let mut enc = Compressor::new(3)?;
let mut out = Vec::new();
let mut buf = vec![0u8; compress_stream_out_size()];

for chunk in [b"first chunk ".as_slice(), b"second chunk".as_slice()] {
    let mut fed = 0;
    while fed < chunk.len() {
        let st = enc.stream(&chunk[fed..], &mut buf, Flush::Continue)?;
        fed += st.input_consumed;
        out.extend_from_slice(&buf[..st.output_produced]);
    }
}
loop {
    let st = enc.stream(&[], &mut buf, Flush::End)?;
    out.extend_from_slice(&buf[..st.output_produced]);
    if st.done {
        break;
    }
}
assert_eq!(rusty_zstd::decompress(&out)?, b"first chunk second chunk");
# Ok::<(), rusty_zstd::Error>(())
```

Dictionaries, trained from your own samples — the win that matters for many small
records:

```rust
use rusty_zstd::{compress_using_dict, decompress_using_dict, train, Dictionary, TrainOptions};

let samples: Vec<Vec<u8>> = (0..256)
    .map(|i| format!("{{\"event\":\"click\",\"id\":{i},\"session\":\"abc123\"}}").into_bytes())
    .collect();
let refs: Vec<&[u8]> = samples.iter().map(|s| s.as_slice()).collect();

let raw = train(&refs, TrainOptions::default())?;    // fastcover, d=8 steps=4
let dict = Dictionary::from_bytes(&raw)?;

let packed = compress_using_dict(&samples[0], &dict, 3)?;
assert_eq!(decompress_using_dict(&packed, &dict)?, samples[0]);
# Ok::<(), rusty_zstd::Error>(())
```

## What is in here

| Area | Surface |
|---|---|
| **One-shot** | `compress`, `decompress`, `decompress_into`, `compress_bound`, `content_size`, `find_frame_compressed_size` |
| **Options** | `CompressOptions`, `AdvancedOptions`, `DecompressOptions`, `CompressionParameters`, `Strategy` |
| **Streaming** | `Compressor`, `Decompressor`, `Flush`, `StreamStatus`, and the four recommended-buffer-size helpers |
| **Dictionaries** | `Dictionary`, `compress_using_dict`, `decompress_using_dict`, `compress_using_prefix` (patch-from), `train` + `TrainOptions` / `TrainAlgo` |
| **Long-range** | `LdmParams`, `DEFAULT_LONG_WINDOW_LOG` |
| **Seekable** | `compress_seekable`, `decompress_frame_at`, `parse_seek_table`, `SeekTable`, `SeekEntry` |
| **Multi-thread** | `compress_mt`, `default_nb_workers`, `resolve_job_size`, `overlap_size` |
| **Inspection** | `get_frame_header`, `FrameHeader`, `FrameKind`, `inspect_frames`, `ListedFrame` |
| **Checksum** | `xxh64` — the frame content hash, usable on its own |

Items marked `#[doc(hidden)]` are campaign instrumentation for the repository's
own benchmark harness. They carry **no semver promise** and may be renamed or
removed in any release.

## Correctness

Gated against facebook/zstd **v1.5.7** as an external process, in both
directions, every commit: C compresses → this decompresses bit-exact, and this
compresses → C's `zstd -t` and `zstd -d` accept it. The XXH64 checksum is gated
against the published vectors, and every SIMD kernel against its scalar twin.

## License

**MIT OR Apache-2.0**, at your option. No GPL/LGPL and no C anywhere in the
dependency tree — CI-enforced with `cargo-deny`.