base-d 0.1.81

Universal multi-alphabet encoder: Encode binary data to 33+ alphabets including RFC standards, hieroglyphs, emoji, and more
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130

# Compression Support Implementation

## Summary

Added comprehensive compression support to base-d, allowing users to compress data before encoding or decompress after decoding. This feature significantly reduces encoded output size for text and repetitive data.

## What Was Added

### 1. Four Compression Algorithms
- **gzip** - General purpose, wide compatibility (default level: 6)
- **zstd** - Fast with excellent ratio (default level: 3)
- **brotli** - Best compression ratio (default level: 6)
- **lz4** - Fastest speed (level: 0)

### 2. New CLI Flags
- `--compress <ALGORITHM>` - Compress before encoding
- `--decompress <ALGORITHM>` - Decompress after decoding
- `--level <N>` - Set compression level
- `--raw` - Output raw binary (no encoding)

### 3. Configuration Support
Added to `dictionaries.toml`:
```toml
[settings]
default_dictionary = "base64"  # Used when compressing without --encode

[compression.gzip]
default_level = 6

[compression.zstd]
default_level = 3

[compression.brotli]
default_level = 6

[compression.lz4]
default_level = 0
```

### 4. New Dependencies
- `flate2` - gzip compression
- `zstd` - Zstandard compression
- `brotli` - Brotli compression
- `lz4` - LZ4 compression

### 5. Library API
Public exports in `lib.rs`:
- `CompressionAlgorithm` enum
- `compress()` function
- `decompress()` function
- `CompressionConfig` struct
- `Settings` struct

### 6. Processing Pipeline
Data flows through stages:
1. **Decode** (if `--decode` specified)
2. **Decompress** (if `--decompress` specified)
3. **Compress** (if `--compress` specified)
4. **Encode** (if `--encode` specified, or default if compressed)

## Usage Examples

### Basic Compression
```bash
# Compress with gzip, encode with base64 (default)
echo "Hello, World!" | base-d --compress gzip
```

### Custom Level
```bash
# Maximum compression with brotli
echo "Data" | base-d --compress brotli --level 11 -e base64
```

### Decompress
```bash
# Decode and decompress
echo "H4sIAAAAAAAA/..." | base-d -d base64 --decompress gzip
```

### Raw Output
```bash
# Output raw compressed binary
echo "Data" | base-d --compress gzip --raw > output.gz
```

### Mix Algorithms
```bash
# Any algorithm with any dictionary
echo "Test" | base-d --compress zstd -e emoji_faces
```

## Files Modified

1. **Cargo.toml** - Added compression dependencies
2. **src/compression.rs** - New module with compression logic
3. **src/lib.rs** - Export compression API
4. **src/core/config.rs** - Added `CompressionConfig` and `Settings`
5. **src/main.rs** - Added CLI flags and pipeline logic
6. **dictionaries.toml** - Added compression config and settings
7. **README.md** - Added compression examples
8. **docs/COMPRESSION.md** - Comprehensive documentation

## Testing

All tests pass:
- 48 unit tests (including 4 new compression roundtrip tests)
- 7 doc tests
- 6 integration tests for compression features

## Performance Impact

- No impact when compression is not used
- Compression adds processing time but significantly reduces output size
- Example: Repetitive text compressed with gzip can be 80%+ smaller

## Backward Compatibility

✓ Fully backward compatible - no breaking changes
- Existing functionality unchanged
- New flags are optional
- Default behavior preserved

## Future Enhancements

Potential improvements (add to ROADMAP.md):
- Streaming mode support for compression
- Additional algorithms (snappy, lzma)
- Parallel compression for large files
- Auto-detection of compressed data format