blockchain-compression 0.1.0

High-performance compression library optimized for blockchain data structures
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91

# Architecture

## Module Structure

```
blockchain-compression/
  src/
    lib.rs              # Crate root, re-exports
    core/
      traits.rs         # CompressionStrategy and related traits
      pattern_engine.rs # Generic pattern recognition engine
    algorithms/
      enhanced_ctw.rs   # Context Tree Weighting compression
      multi_pass.rs     # Multi-pass iterative compression
      practical_max.rs  # Combined maximum compression
    presets/
      solana.rs         # Solana-optimized compressor and presets
```

## Design Principles

### Trait-Based Composition

The library is built around the `CompressionStrategy` trait. Every compressor -- whether it's the Solana preset, the pattern engine, or the CTW algorithm -- implements this trait. This makes compressors interchangeable and composable.

```
CompressionStrategy (core trait)
  |
  +-- PatternCompressionStrategy  (pattern management)
  +-- PipelineCompressionStrategy (multi-stage pipelines)
  +-- AdaptiveCompressionStrategy (learning-based)
```

### Three Layers

**Core** (`src/core/`) defines traits and the generic pattern recognition engine. It has no blockchain-specific knowledge.

**Algorithms** (`src/algorithms/`) implements compression algorithms:

- `EnhancedCTW` -- prediction-based compression using context tree weighting
- `MultiPassCompressor` -- applies compression in multiple passes, stopping when improvement drops below a threshold
- `PracticalMaxCompression` -- orchestrates pattern engine + CTW + multi-pass for maximum compression

**Presets** (`src/presets/`) provides blockchain-specific configurations. The `SolanaCompressor` wraps Zstandard with custom dictionaries built from common Solana byte patterns.

### Custom Dictionary Approach

The Solana compressor builds a custom Zstandard dictionary at initialization from hardcoded patterns:

1. Common program IDs (System, Token, Associated Token, Serum DEX, etc.)
2. Instruction discriminators (Transfer, Initialize, Close, Approve)
3. Common token amounts in lamports (0.01, 0.1, 1, 10 SOL)
4. Transaction structure markers (signature counts)
5. Base58 character set

This dictionary is passed to the Zstandard encoder/decoder, allowing it to reference these common byte sequences during compression without storing them in the output.

### Compression Flow

```
Input Data
    |
    v
SolanaCompressor.compress()
    |
    +-- Select compression level from preset
    +-- Load custom Solana dictionary
    +-- Encode via zstd::Encoder with dictionary
    +-- Update compression statistics
    |
    v
Compressed Bytes
```

Decompression reverses the process using the same dictionary.

### Error Handling

All operations return `Result<_, CompressionError>`. The error type is an enum with variants for each failure category (format, configuration, I/O, internal). Errors carry descriptive messages for debugging.

### Feature Flags

Compression backends are behind feature flags to avoid pulling in unnecessary dependencies:

| Flag | Crate | Description |
|------|-------|-------------|
| `deflate` (default) | `flate2` | DEFLATE compression |
| `zstd` | `zstd` | Zstandard compression |
| `lz4` | `lz4_flex` | LZ4 compression |

The Solana presets require the `zstd` feature. The pattern engine can use any backend.