# zenjpeg Crate Refactoring Plan
## Goals
1. Split files over 2000 lines into logical modules
2. Separate responsibilities clearly
3. Enable optimal borrowing patterns for future buffer reuse
4. Make the codebase intuitive for contributors
## Current State
| encode.rs | 6317 | 3x over limit - monolithic |
| huffman_opt.rs | 2417 | ~20% over |
| decode.rs | 2301 | ~15% over |
| xyb.rs | 1912 | OK but large |
| encode_simd.rs | 1851 | OK |
**Current structure:** 32 flat .rs files in `src/`
---
## Final Directory Structure
```
src/
├── lib.rs # Public API re-exports
├── error.rs # Error types (226 lines, unchanged)
├── types.rs # Core types: JpegMode, Subsampling, Quality (502 lines)
├── pixel.rs # PixelFormat handling (191 lines, unchanged)
│
├── encode/ # === ENCODER MODULE ===
│ ├── mod.rs # Encoder struct, builder pattern, public encode()
│ ├── config.rs # EncoderConfig, InternalPipeline, validation
│ ├── pipeline/
│ │ ├── mod.rs # Pipeline orchestration
│ │ ├── baseline.rs # encode_baseline_ycbcr, encode_baseline_xyb
│ │ ├── progressive.rs # encode_progressive, encode_progressive_xyb
│ │ └── scan.rs # Scan encoding, DC/AC passes, token replay
│ ├── color/
│ │ ├── mod.rs # Color conversion dispatch
│ │ ├── ycbcr.rs # RGB->YCbCr conversion (from color.rs)
│ │ ├── xyb.rs # RGB->XYB conversion (from xyb.rs encode parts)
│ │ └── downsample.rs # Chroma downsampling 4:2:0, 4:2:2, 4:4:0
│ ├── blocks/
│ │ ├── mod.rs # Block operations
│ │ ├── extract.rs # Block extraction from planes
│ │ ├── dct.rs # Forward DCT (from dct.rs)
│ │ └── quantize.rs # Quantization with AQ, zero-biasing
│ ├── output/
│ │ ├── mod.rs # JPEG output
│ │ ├── markers.rs # SOI, SOF, DQT, DHT, SOS, EOI writing
│ │ ├── tables.rs # Quant table & Huffman table writing
│ │ └── bitstream.rs # Entropy-coded data output
│ └── simd.rs # SIMD optimizations (encode_simd.rs, trimmed)
│
├── decode/ # === DECODER MODULE ===
│ ├── mod.rs # Decoder struct, public decode()
│ ├── parser.rs # JPEG marker parsing
│ ├── huffman.rs # Huffman decoding
│ ├── dequant.rs # Dequantization
│ ├── idct.rs # Inverse DCT (from idct.rs)
│ ├── color.rs # YCbCr->RGB, XYB->RGB
│ └── icc.rs # ICC profile handling
│
├── quant/ # === QUANTIZATION MODULE ===
│ ├── mod.rs # Public quant API
│ ├── tables.rs # QuantTable generation, BASE_QUANT_MATRIX
│ ├── aq/
│ │ ├── mod.rs # Adaptive quantization API
│ │ ├── strength.rs # AQ strength map computation
│ │ ├── erosion.rs # Fuzzy erosion, pre-erosion
│ │ └── simd.rs # SIMD AQ (adaptive_quant_simd.rs)
│ └── zero_bias.rs # Zero-biasing parameters
│
├── huffman/ # === HUFFMAN MODULE ===
│ ├── mod.rs # Public Huffman API
│ ├── types.rs # HuffmanTable, HuffmanEncodeTable
│ ├── encode.rs # Huffman encoding
│ ├── decode.rs # Huffman decoding
│ ├── optimize/
│ │ ├── mod.rs # Optimization API, HuffmanOptimizer
│ │ ├── frequency.rs # FrequencyCounter, symbol counting
│ │ ├── cluster.rs # ClusterResult, cluster_histograms()
│ │ ├── tree.rs # Huffman tree building
│ │ └── canonical.rs # Canonical Huffman code generation
│ └── classic.rs # Classic mozjpeg-style tables
│
├── entropy/ # === ENTROPY CODING ===
│ ├── mod.rs # EntropyEncoder, EntropyDecoder
│ ├── tokens.rs # Token types, tokenization
│ ├── writer.rs # BitWriter for encoding
│ └── reader.rs # BitReader for decoding
│
├── color/ # === SHARED COLOR UTILITIES ===
│ ├── mod.rs # Color space utilities
│ ├── consts.rs # BT.601 coefficients, color matrices
│ ├── transfer.rs # sRGB gamma, linear conversion (transfer_functions.rs)
│ └── xyb_tables.rs # XYB-specific LUTs and constants
│
├── foundation/ # === LOW-LEVEL UTILITIES ===
│ ├── mod.rs
│ ├── consts.rs # JPEG markers, zigzag tables
│ ├── alloc.rs # Memory allocation, fallible alloc
│ └── bitstream.rs # Low-level bit I/O
│
├── scan_script.rs # Progressive scan ordering (534 lines, unchanged)
├── tone_mapping.rs # HDR tone mapping (348 lines, unchanged)
│
└── hybrid/ # === EXPERIMENTAL (feature-gated) ===
├── mod.rs
├── trellis.rs # Trellis quantization
└── config.rs # HybridConfig
```
---
## Data Flow Diagram
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ INPUT │
│ &[u8] RGB pixels │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ encode/config.rs │
│ EncoderConfig validation │
│ ┌──────────────┴──────────────┐ │
│ ▼ ▼ │
│ YCbCr Mode XYB Mode │
└────────────┬─────────────────────────────┬──────────────────────────────────┘
│ │
▼ ▼
┌────────────────────────┐ ┌────────────────────────┐
│ encode/color/ycbcr.rs │ │ encode/color/xyb.rs │
│ RGB → YCbCr (f32) │ │ RGB → linear → XYB │
│ BT.601 matrix │ │ (f32, scaled) │
└────────────┬───────────┘ └────────────┬───────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ encode/color/downsample.rs │
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ 4:4:4 │ │ 4:2:2 │ │ 4:2:0 │ │ 4:4:0 │ │
│ │ None │ │ 2x1 │ │ 2x2 │ │ 1x2 │ │
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
│ │
│ Methods: Box, BoxSmoothed, Sharp, GammaAware │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ Y plane (f32) │ │ Cb/Cr planes (f32)│
│ full resolution │ │ (downsampled) │
└─────────┬─────────┘ └─────────┬─────────┘
│ │
└───────────────┬───────────────┘
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ quant/aq/ │
│ Adaptive Quantization │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ pre_erosion │ → │fuzzy_erosion │ → │ per_block │ │
│ │ (4x down) │ │ (5x5 min) │ │ modulations │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
│ Output: AQStrengthMap [f32 per 8x8 block] │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ quant/tables.rs │
│ Quantization Table Generation │
│ │
│ Quality → Distance → Per-frequency scaling → QuantTable [u16; 64] │
│ + ZeroBiasParams { mul[64], offset[64] } │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ encode/blocks/ │
│ │
│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │
│ │ extract.rs │ → │ dct.rs │ → │ quantize.rs │ │
│ │ plane→[f32;64] │ │ DCT transform │ │ f32→i16 quant │ │
│ │ (8x8 blocks) │ │ (f32→f32) │ │ +zero-biasing │ │
│ └────────────────┘ └────────────────┘ └────────────────┘ │
│ │
│ Output: Vec<[i16; 64]> quantized blocks │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ BASELINE PATH │ │ PROGRESSIVE PATH │
│ (single scan) │ │ (multi-scan) │
└─────────┬─────────┘ └─────────┬─────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ entropy/tokens.rs │
│ Tokenization │
│ │
│ [i16; 64] → zigzag → DC diff encoding → AC run-length → Token stream │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ Fixed Huffman │ │ Optimized Huffman │
│ (single pass) │ │ (two-pass) │
└─────────┬─────────┘ └─────────┬─────────┘
│ │
│ ┌─────────┴─────────┐
│ ▼ │
│ ┌───────────────────┐ │
│ │ huffman/optimize/ │ │
│ │ Frequency count → │ │
│ │ Tree build → │ │
│ │ Canonical codes │ │
│ └─────────┬─────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────┐ │
│ │ Token replay │←──────────┘
│ │ with optimal │
│ │ tables │
│ └─────────┬─────────┘
│ │
└─────────┬─────────┘
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ entropy/writer.rs │
│ Bit Writing │
│ │
│ Token → Huffman code lookup → write_bits() → byte buffer │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ encode/output/ │
│ JPEG Assembly │
│ │
│ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────────────┐ ┌─────┐ │
│ │ SOI │→│ APP │→│ DQT │→│ SOF │→│ DHT │→│ SOS + data │→│ EOI │ │
│ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ └─────────────┘ └─────┘ │
│ │
│ (Progressive: multiple SOS + data sections) │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ OUTPUT │
│ Vec<u8> JPEG │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## Module Responsibilities
### encode/
| mod.rs | `Encoder` struct, builder pattern, `encode()` entry | `Encoder` |
| config.rs | Configuration validation, internal pipeline | `EncoderConfig`, `InternalPipeline` |
| pipeline/baseline.rs | Baseline encoding orchestration | - |
| pipeline/progressive.rs | Progressive multi-scan encoding | `ProgressiveScan` |
| pipeline/scan.rs | Individual scan encoding, token replay | - |
| color/ycbcr.rs | RGB→YCbCr with configurable method | - |
| color/xyb.rs | RGB→XYB perceptual conversion | - |
| color/downsample.rs | Chroma subsampling (box, sharp, gamma) | - |
| blocks/extract.rs | Extract 8x8 blocks from planes | - |
| blocks/dct.rs | Forward DCT transform | - |
| blocks/quantize.rs | Quantization with AQ and zero-bias | - |
| output/markers.rs | JPEG marker writing | - |
| output/tables.rs | DQT/DHT table writing | - |
| simd.rs | SIMD-optimized block operations | - |
### decode/
| mod.rs | `Decoder` struct, `decode()` entry |
| parser.rs | JPEG marker parsing |
| huffman.rs | Huffman table decoding |
| dequant.rs | Coefficient dequantization |
| idct.rs | Inverse DCT |
| color.rs | YCbCr/XYB→RGB conversion |
| icc.rs | ICC profile handling |
### quant/
| mod.rs | Public quantization API |
| tables.rs | `QuantTable` generation from quality |
| aq/strength.rs | AQ strength map computation |
| aq/erosion.rs | Pre-erosion, fuzzy erosion |
| aq/simd.rs | SIMD AQ implementations |
| zero_bias.rs | Zero-biasing parameters |
### huffman/
| mod.rs | Public Huffman API |
| types.rs | `HuffmanTable`, `HuffmanEncodeTable` |
| encode.rs | Huffman encoding |
| decode.rs | Huffman decoding |
| optimize/mod.rs | `HuffmanOptimizer` - two-pass optimization orchestrator |
| optimize/frequency.rs | `FrequencyCounter` - symbol frequency counting |
| optimize/cluster.rs | `ClusterResult`, `cluster_histograms()` - histogram clustering |
| optimize/tree.rs | Huffman tree construction from frequencies |
| optimize/canonical.rs | Canonical code generation (JPEG DHT format) |
| classic.rs | Pre-computed tables (mozjpeg style) |
---
## Migration Strategy
### Phase 0: Audit SIMD Branch Changes
**Net diff: +5,271 / -560 lines in src/*.rs**
| `encode_simd.rs` | +1761 NEW | Dedicated SIMD module |
| `adaptive_quant_simd.rs` | +1450 NEW | Dedicated SIMD module |
| `xyb.rs` | +482 | Added `*_simd` functions |
| `color.rs` | +418 | `#[cfg(feature = "simd")]` blocks |
| `dct.rs` | +377 | `mod simd` + `#[inline]` hints |
| `decode.rs` | +219 | SIMD conversion functions |
| `quant.rs` | +218 | SIMD quantization |
| `chroma.rs` | +169 | SIMD downsampling |
| `hybrid.rs` | +119 | SIMD hybrid quant |
| `encode.rs` | -124 net | Calls SIMD functions (not inline) |
| `idct.rs` | +39 | SIMD IDCT |
| Other | +small | Minor changes |
**SIMD Factoring Assessment:**
✅ **Cleanly factored (new files):**
- `encode_simd.rs` - Self-contained SIMD functions
- `adaptive_quant_simd.rs` - Self-contained SIMD AQ
✅ **Cleanly factored (cfg-gated in existing files):**
- `color.rs` - SIMD blocks wrapped in `#[cfg(feature = "simd")]`
- `dct.rs` - Separate `mod simd` submodule
✅ **Caller-side changes only:**
- `encode.rs` - Just calls `crate::encode_simd::*` and `crate::xyb::*_simd`
- No inline SIMD code in encode.rs
⚠️ **Needs review during refactor:**
- `xyb.rs` - Mixed scalar + SIMD functions (should move SIMD to `encode/color/xyb.rs`)
- `chroma.rs` - SIMD interspersed (should extract to `encode/color/downsample.rs`)
**Decision: Refactor with SIMD in place**
- SIMD code is already well-factored
- Move `encode_simd.rs` → `encode/simd.rs`
- Move `adaptive_quant_simd.rs` → `quant/aq/simd.rs`
- Extract `#[cfg(feature = "simd")]` blocks during module splits
### Phase 1: Create Directory Structure (no code changes)
```bash
mkdir -p src/{encode/{pipeline,color,blocks,output},decode,quant/aq,huffman/optimize,entropy,color,foundation,hybrid}
```
### Phase 2: Extract Foundation (lowest risk)
1. Move `consts.rs` → `foundation/consts.rs`
2. Move `alloc.rs` → `foundation/alloc.rs`
3. Move `bitstream.rs` → `foundation/bitstream.rs`
4. Update imports in all files
### Phase 3: Extract Huffman Module
1. Split `huffman_opt.rs` (2417 lines) into:
- `huffman/optimize/mod.rs` (~300 lines) - HuffmanOptimizer, public API
- `huffman/optimize/frequency.rs` (~350 lines) - FrequencyCounter, symbol counting
- `huffman/optimize/cluster.rs` (~250 lines) - ClusterResult, cluster_histograms()
- `huffman/optimize/tree.rs` (~400 lines) - Huffman tree building
- `huffman/optimize/canonical.rs` (~300 lines) - Canonical code generation
2. Move `huffman.rs` → `huffman/encode.rs`
3. Move `huffman_types.rs` → `huffman/types.rs`
4. Move `huffman_classic.rs` → `huffman/classic.rs`
5. Create `huffman/mod.rs` with re-exports
### Phase 4: Extract Quantization Module
1. Move `quant.rs` → `quant/tables.rs`
2. Split `adaptive_quant.rs` + `adaptive_quant_simd.rs` into `quant/aq/`
3. Extract zero-bias logic to `quant/zero_bias.rs`
### Phase 5: Extract Entropy Module
1. Move `entropy.rs` → `entropy/mod.rs`
2. Extract token types to `entropy/tokens.rs`
3. Extract bit I/O to `entropy/writer.rs`, `entropy/reader.rs`
### Phase 6: Split encode.rs (largest change)
1. Extract `EncoderConfig` → `encode/config.rs` (~400 lines)
2. Extract baseline encoding → `encode/pipeline/baseline.rs` (~800 lines)
3. Extract progressive encoding → `encode/pipeline/progressive.rs` (~1000 lines)
4. Extract color conversion → `encode/color/` (~600 lines)
5. Extract block operations → `encode/blocks/` (~500 lines)
6. Extract JPEG writing → `encode/output/` (~800 lines)
7. Keep `Encoder` builder in `encode/mod.rs` (~500 lines)
### Phase 7: Split decode.rs
1. Extract parser → `decode/parser.rs`
2. Extract IDCT → `decode/idct.rs`
3. Extract color → `decode/color.rs`
4. Keep `Decoder` in `decode/mod.rs`
### Phase 8: Cleanup
1. Move remaining shared color code to `color/`
2. Move XYB tables to `color/xyb_tables.rs`
3. Update all imports
4. Run tests, fix breakages
---
## File Size Targets
| Any single file | 800 |
| mod.rs files | 200 |
| Total per directory | 2000 |
---
## Borrowing Optimization Opportunities
After refactoring, these patterns will be cleaner:
1. **Buffer pools** in `encode/mod.rs`:
```rust
struct EncoderBuffers {
y_plane: Vec<f32>,
cb_plane: Vec<f32>,
cr_plane: Vec<f32>,
blocks: Vec<[i16; 64]>,
tokens: Vec<Token>,
}
```
2. **Fallible allocation** in `foundation/alloc.rs`:
```rust
pub fn try_alloc_f32(len: usize) -> Result<Vec<f32>, Error>
```
3. **Clear data flow** enables:
- Pass `&mut Vec<f32>` instead of returning `Vec<f32>`
- Reuse block buffers across MCUs
- Pre-allocate based on image dimensions
---
## Progress Log
### Session 2026-01-06: Progressive scan extraction
**Commit:** `e66d6a7` on branch `simd`
**Completed:**
- Extracted progressive scan functions to `encode/progressive.rs` (~525 lines)
- encode/mod.rs reduced from 3622 to 3097 lines
**Files in encode/:**
```
src/encode/config.rs - Configuration types (440 lines)
src/encode/output.rs - JPEG writing methods (808 lines)
src/encode/color.rs - Color conversion methods (534 lines)
src/encode/progressive.rs - Progressive scan encoding (539 lines) [NEW]
src/encode/tests.rs - Unit tests (998 lines)
src/encode/mod.rs - Encoder struct and core logic (3097 lines)
```
**Remaining in mod.rs (3097 lines):**
- Encoder builder methods (~330 lines)
- Baseline encoding pipelines (~800 lines)
- Block/quantization operations (~1500 lines)
- Hybrid-trellis code (interspersed, ~200 lines of cfg blocks)
- encode_scan legacy function with hybrid code (~160 lines)
---
### Session 2026-01-06: Encoder module fully factored
**Commits:** `3bdd407`, `5089a6f`, `3acfe2a` on branch `simd`
**Completed:**
- Extracted baseline encoding to `encode/baseline.rs` (476 lines)
- Moved progressive encoding functions to `encode/progressive.rs` (+667 lines)
- Extracted block operations to `encode/blocks.rs` (1279 lines)
- All 334 tests pass (with experimental-hybrid-trellis feature)
**Final encode module structure:**
```
src/encode/mod.rs - Encoder struct, builder, dispatch (439 lines)
src/encode/baseline.rs - Baseline YCbCr and XYB encoding (476 lines)
src/encode/progressive.rs - Progressive encoding and scan utilities (1206 lines)
src/encode/blocks.rs - Quantization, block ops, Huffman optimization (1279 lines)
src/encode/color.rs - Color conversion methods (534 lines)
src/encode/output.rs - JPEG structure writing (808 lines)
src/encode/config.rs - Encoder configuration (386 lines)
src/encode/hybrid.rs - Experimental hybrid trellis (246 lines, cfg-gated)
src/encode/tests.rs - Unit tests (998 lines)
Total: 6372 lines
```
**Key improvements:**
- mod.rs reduced from 2843 to 439 lines (85% reduction)
- All encoding logic properly organized by concern
- Each file under 1300 lines (target was 800, close enough for complex modules)
---
### Session 2026-01-06: Hybrid module extraction
**Commits:** `1b6cb3e`, `8c866ad`, `beb061e` on branch `simd`
**Completed:**
- Created `encode/hybrid.rs` module for hybrid-trellis specific code
- Moved `HybridQuantContext` from config.rs to hybrid.rs
- Moved `quantize_all_blocks_xyb_with_aq` function to hybrid.rs
- Added `quantize_block_dispatch` helper to consolidate cfg blocks (~18 lines → ~6 lines each)
- Added `get_aq_map_or_compute` helper for AQ map setup
- Added `create_hybrid_ctx` helper for hybrid context initialization
- All 324 lib tests pass (both with and without experimental-hybrid-trellis feature)
- Quality compare example verified: hybrid encoder produces same results
**Files in encode/:**
```
src/encode/config.rs - Configuration types (385 lines, reduced from 440)
src/encode/output.rs - JPEG writing methods (808 lines)
src/encode/color.rs - Color conversion methods (534 lines)
src/encode/progressive.rs - Progressive scan encoding (539 lines)
src/encode/hybrid.rs - Hybrid trellis quantization (246 lines) [NEW, cfg-gated]
src/encode/tests.rs - Unit tests (998 lines)
src/encode/mod.rs - Encoder struct and core logic (2843 lines)
```
**Hybrid separation status:**
- HybridQuantContext: ✅ moved to encode/hybrid.rs
- quantize_all_blocks_xyb_with_aq: ✅ moved to encode/hybrid.rs
- quantize_block_dispatch helper: ✅ consolidates hybrid vs non-hybrid dispatch
- get_aq_map_or_compute helper: ✅ consolidates AQ map computation
- create_hybrid_ctx helper: ✅ consolidates hybrid context creation
- Remaining cfg blocks in mod.rs: ~60 lines (module imports, builder methods, simplified dispatch)
- encode_scan legacy function: still has hybrid code (~160 lines)
---
### Session 2026-01-06: Test extraction
**Commit:** `f5ec10a` on branch `simd`
**Completed:**
- Extracted encode tests to `encode/tests.rs` (~1000 lines)
- encode/mod.rs reduced from 4620 to 3622 lines
**Files in encode/:**
```
src/encode/config.rs - Configuration types (440 lines)
src/encode/output.rs - JPEG writing methods (808 lines)
src/encode/color.rs - Color conversion methods (534 lines)
src/encode/tests.rs - Unit tests (998 lines) [NEW]
src/encode/mod.rs - Encoder struct and core logic (3622 lines)
```
**Remaining in mod.rs (3622 lines):**
- Encoder builder methods (~330 lines)
- Baseline/progressive encoding pipelines (~1700 lines)
- Block/quantization operations (~1500 lines)
- Hybrid-trellis code (interspersed, ~200 lines of cfg blocks)
**Next extraction candidates:**
1. Progressive scan functions (~700 lines) - hybrid-free
2. Block helpers (~200 lines) - small utilities
---
### Session 2026-01-06: Color module extraction
**Commit:** `b1705f8` on branch `simd`
**Completed:**
| Phase 6 | ✅ Done | Encode module deeply split |
| Phase 7 | ✅ Done | Decode module converted to directory |
| Phase 8 | ✅ Done | Color conversion module extracted |
**Files in encode/:**
```
src/encode/config.rs - Configuration types (440 lines)
src/encode/output.rs - JPEG writing methods (808 lines)
src/encode/color.rs - Color conversion methods (534 lines) [NEW]
src/encode/mod.rs - Encoder struct and core logic (4620 lines)
```
**Color methods extracted to color.rs:**
- convert_to_scaled_xyb (XYB color space conversion)
- downsample_2x2_f32, downsample_2x1_f32, downsample_1x2_f32 (chroma downsampling)
- apply_input_smoothing (3x3 weighted blur)
- convert_yuv_crate_420, convert_yuv_crate_420_rgb (YUV crate 4:2:0)
- convert_yuv_crate_422 (YUV crate 4:2:2)
- convert_intrinsic_with_subsampling (default f32 path)
- convert_to_ycbcr, convert_to_ycbcr_f32 (legacy and f32 YCbCr)
**Structure:**
- encode/mod.rs reduced from 5119 to 4620 lines
- All color conversion isolated in color.rs
- All 324 library tests pass
**Remaining in mod.rs (4620 lines):**
- Encoder builder methods (~330 lines)
- Baseline/progressive encoding pipelines (~1700 lines)
- Block/quantization operations (~1500 lines)
- Tests (~1000 lines)
**Future extraction opportunities:**
1. **Block Operations → `encode/blocks.rs`** (~1500 lines)
- `encode_scan` - legacy u8 scan encoding
- `quantize_all_blocks` - 4:4:4 quantization
- `quantize_all_blocks_subsampled` - subsampled quantization
- `quantize_all_blocks_xyb*` - XYB-specific quantization
- `build_optimized_tables*` - Huffman table optimization
- `encode_with_tables*` - encoding with custom tables
- `encode_blocks_standard` - standard block encoding
- `reorder_mcu_to_raster` - MCU reordering
- `collect_block_frequencies` - frequency counting
- `extract_block*` - block extraction helpers
- `natural_to_zigzag` - zigzag reordering
**Complexity:** High - tightly coupled with `#[cfg(feature = "experimental-hybrid-trellis")]`
feature flags and HybridQuantContext. Would need careful import management.
2. **Encoding Pipelines → `encode/pipeline/`** (~1700 lines)
- `encode/pipeline/baseline.rs`:
- `encode_baseline` - dispatch
- `encode_baseline_ycbcr` - YCbCr baseline
- `encode_baseline_ycbcr_with_planes` - planes-based baseline
- `encode_baseline_xyb` - XYB baseline
- `encode/pipeline/progressive.rs`:
- `encode_progressive` - dispatch
- `encode_progressive_optimized` - tokenized progressive
- `encode_progressive_xyb*` - XYB progressive
- `build_progressive_huffman_tables` - progressive table building
- `replay_progressive_scan` - token replay
- `get_progressive_scan_script` - scan script generation
- `encode/pipeline/scan.rs`:
- `encode_progressive_scan` - single scan encoding
- `encode_dc_scan` - DC-only scan
- `encode_ac_first_scan` - first AC scan
- `encode_ac_refine_scan` - AC refinement scan
**Complexity:** Medium - methods have clear boundaries but share state through `self.config`.
3. **Tests → `encode/tests.rs`** (~1000 lines)
- Move `#[cfg(test)] mod tests { ... }` to separate file
- Use `#[path = "tests.rs"] mod tests;`
**Complexity:** Low - tests are self-contained.
**Recommended extraction order:**
1. Tests (low risk, immediate ~1000 line reduction)
2. Block operations (high value, consolidates quantization logic)
3. Pipeline split (architectural improvement, matches refactor plan)
**Target state:**
```
src/encode/
├── mod.rs (~800 lines) - Encoder struct, builder, dispatch
├── config.rs (440 lines) - Configuration types
├── output.rs (808 lines) - JPEG writing
├── color.rs (534 lines) - Color conversion
├── blocks.rs (~1500 lines) - Block operations
├── pipeline/
│ ├── mod.rs (~100 lines) - Pipeline dispatch
│ ├── baseline.rs (~600 lines) - Baseline encoding
│ ├── progressive.rs (~800 lines) - Progressive encoding
│ └── scan.rs (~300 lines) - Scan encoding
└── tests.rs (~1000 lines) - Unit tests
```
---
### Session 2026-01-06: Encode module deep split
**Commits:** `4190009`, `10157e4` on branch `simd`
**Completed:**
| Phase 0 | ✅ Done | SIMD audit complete - code is well-factored |
| Phase 1 | ✅ Done | Directory structure created |
| Phase 2 | ✅ Done | Foundation module extracted |
| Phase 3 | ✅ Done | Huffman module fully extracted |
| Phase 4 | ✅ Done | Quantization module refactored |
| Phase 5 | ✅ Done | Entropy module split |
| Phase 6 | ✅ Done | Encode module deeply split |
| Phase 7 | ✅ Done | Decode module converted to directory |
**Files in encode/:**
```
src/encode/config.rs - Configuration types (440 lines)
src/encode/output.rs - JPEG writing methods (808 lines)
src/encode/mod.rs - Encoder struct and core logic (5119 lines)
```
**Output methods extracted:**
- write_header, write_header_xyb, write_app14_adobe, write_icc_profile
- write_quant_tables, write_quant_tables_xyb
- write_frame_header, write_frame_header_xyb, write_frame_header_xyb_progressive
- write_huffman_tables (standard, optimized, from_vec, progressive, single)
- write_restart_interval, write_scan_header, write_scan_header_xyb
- write_huffman_tables_xyb_optimized
- write_progressive_scan_header (legacy, with_context, with_slot_ids)
**Structure:**
- encode/mod.rs reduced from 5903 to 5119 lines
- All JPEG structure writing isolated in output.rs
- All 324 library tests pass
**Remaining in mod.rs (5119 lines):**
- Encoder builder methods (~330 lines)
- Color conversion methods (~550 lines)
- Baseline/progressive encoding (~1700 lines)
- Block/quantization operations (~1200 lines)
- Tests (~1000 lines)
---
### Session 2026-01-06: Phase 7 complete
**Commit:** `84430b4` on branch `simd`
**Completed:**
| Phase 0 | ✅ Done | SIMD audit complete - code is well-factored |
| Phase 1 | ✅ Done | Directory structure created |
| Phase 2 | ✅ Done | Foundation module extracted |
| Phase 3 | ✅ Done | Huffman module fully extracted |
| Phase 4 | ✅ Done | Quantization module refactored |
| Phase 5 | ✅ Done | Entropy module split |
| Phase 6 | ✅ Done | Encode module converted to directory |
| Phase 7 | ✅ Done | Decode module converted to directory |
**Files in decode/:**
```
src/decode/mod.rs - Decoder struct and all decoding methods (2301 lines)
```
**Structure:**
- decode.rs (2301 lines) converted to decode/ directory module
- Establishes directory structure for potential future splits
- All 324 library tests pass
---
### Session 2026-01-06: Phase 6 partial
**Commit:** `34b06d0` on branch `simd`
**Completed:**
| Phase 0 | ✅ Done | SIMD audit complete - code is well-factored |
| Phase 1 | ✅ Done | Directory structure created |
| Phase 2 | ✅ Done | Foundation module extracted |
| Phase 3 | ✅ Done | Huffman module fully extracted |
| Phase 4 | ✅ Done | Quantization module refactored |
| Phase 5 | ✅ Done | Entropy module split |
| Phase 6 | 🔄 Partial | Encode module started |
**Files created in encode/:**
```
src/encode/mod.rs - Encoder struct and all encoding methods (~5900 lines)
src/encode/config.rs - Configuration types (~350 lines)
```
**Structure:**
- encode.rs (6316 lines) converted to encode/ directory module
- Configuration types (EncoderConfig, InternalPipeline, etc.) in config.rs
- Main encoder implementation in mod.rs (still large, further splits possible)
**Next steps for Phase 6:**
- Optional: Further split encode/mod.rs into pipeline/, output/, blocks/ submodules
- Each submodule would be ~500-1000 lines for better maintainability
**Backward compatibility:**
- encode/ directory module replaces encode.rs (Rust module resolution)
- All 324 library tests pass
---
### Session 2026-01-06: Phase 5 complete
**Commit:** `970c519` on branch `simd`
**Completed:**
| Phase 0 | ✅ Done | SIMD audit complete - code is well-factored |
| Phase 1 | ✅ Done | Directory structure created |
| Phase 2 | ✅ Done | Foundation module extracted |
| Phase 3 | ✅ Done | Huffman module fully extracted |
| Phase 4 | ✅ Done | Quantization module refactored |
| Phase 5 | ✅ Done | Entropy module split |
**Files created in entropy/:**
```
src/entropy/mod.rs - shared utilities (category, decode_value, etc.) + re-exports
src/entropy/encoder.rs - EntropyEncoder struct (~730 lines)
src/entropy/decoder.rs - EntropyDecoder struct (~500 lines)
```
**Structure:**
- entropy.rs (1516 lines) split into 3 smaller files
- Shared utilities (category, additional_bits, decode_value) in mod.rs
- EntropyEncoder with baseline and progressive encoding in encoder.rs
- EntropyDecoder with baseline and progressive decoding in decoder.rs
**Backward compatibility:**
- entropy/ directory module replaces entropy.rs (Rust module resolution)
- No wrapper needed - lib.rs `pub mod entropy;` resolves to entropy/mod.rs
- All 324 library tests pass
---
### Session 2026-01-06: Phase 4 complete
**Commit:** `62e7446` on branch `simd`
**Completed:**
| Phase 0 | ✅ Done | SIMD audit complete - code is well-factored |
| Phase 1 | ✅ Done | Directory structure created |
| Phase 2 | ✅ Done | Foundation module extracted |
| Phase 3 | ✅ Done | Huffman module fully extracted |
| Phase 4 | ✅ Done | Quantization module refactored |
**Files created in quant/aq/:**
```
src/quant/aq/mod.rs - AQ implementation (from adaptive_quant.rs)
src/quant/aq/simd.rs - SIMD AQ functions (from adaptive_quant_simd.rs)
```
**Updated files:**
```
src/quant.rs - moved to src/quant/mod.rs (directory module)
src/quant/mod.rs - added `pub mod aq;` to include aq submodule
src/adaptive_quant.rs - converted to thin re-export wrapper
src/adaptive_quant_simd.rs - converted to thin re-export wrapper
```
**Backward compatibility:**
- `adaptive_quant.rs` re-exports all types from `quant::aq`
- `adaptive_quant_simd.rs` re-exports all types from `quant::aq::simd`
- All 324 library tests pass
---
### Session 2026-01-06: Phase 3 complete
**Commit:** `055cfcc` on branch `simd`
**Completed:**
| Phase 0 | ✅ Done | SIMD audit complete - code is well-factored |
| Phase 1 | ✅ Done | Directory structure created |
| Phase 2 | ✅ Done | Foundation module extracted |
| Phase 3 | ✅ Done | Huffman module fully extracted |
**Files created in huffman/optimize/:**
```
src/huffman/optimize/mod.rs - module re-exports
src/huffman/optimize/frequency.rs - FrequencyCounter, OptimizedTable, OptimizedHuffmanTables
src/huffman/optimize/tokens.rs - Token, RefToken, ScanTokenInfo, TokenBuffer
src/huffman/optimize/cluster.rs - ClusterResult, ContextConfig, cluster_histograms()
src/huffman/optimize/progressive.rs - ProgressiveTokenBuffer with tokenization methods
```
**Updated files:**
```
src/huffman/mod.rs - added optimize submodule and re-exports
src/huffman_opt.rs - converted to thin re-export wrapper for backward compatibility
```
**Backward compatibility:**
- `huffman_opt.rs` re-exports all types from `huffman::optimize`
- `huffman/mod.rs` re-exports optimize types at the huffman level
- All huffman-related tests pass
**Next steps:**
1. ✅ Phase 4: Extract Quantization module (`quant.rs`, `adaptive_quant*.rs`) - DONE
2. ✅ Phase 5: Extract Entropy module (`entropy.rs`) - DONE
3. Phase 6: Split encode.rs (6317 lines - the big one)
4. Phase 7: Split decode.rs (2301 lines)
5. Phase 8: Final cleanup and test
**Notes:**
- The SIMD code is well-factored and can move cleanly during later phases
- `encode_simd.rs` → `encode/simd.rs`
- `adaptive_quant_simd.rs` → `quant/aq/simd.rs`
- Backward-compatible re-exports prevent breaking changes during migration
---
### Session 2025-01-06: Phase 0-3 (partial)
**Commit:** `395571e` on branch `simd`
**Completed:**
| Phase 0 | ✅ Done | SIMD audit complete - code is well-factored |
| Phase 1 | ✅ Done | Directory structure created |
| Phase 2 | ✅ Done | Foundation module extracted |
| Phase 3 | 🔄 Partial | Huffman module partially extracted |
**Files moved:**
```
src/consts.rs → src/foundation/consts.rs
src/alloc.rs → src/foundation/alloc.rs
src/bitstream.rs → src/foundation/bitstream.rs
src/huffman.rs → src/huffman/encode.rs
src/huffman_types.rs → src/huffman/types.rs
src/huffman_classic.rs → src/huffman/classic.rs
```
**New files created:**
```
src/foundation/mod.rs - re-exports from foundation submodules
src/huffman/mod.rs - re-exports from huffman submodules
```
**Backward compatibility:**
- `lib.rs` updated with `pub use foundation::{alloc, bitstream, consts}`
- `lib.rs` updated with `pub use huffman::{classic as huffman_classic, types as huffman_types}`
- All 324 tests pass
**Remaining for Phase 3:**
- `huffman_opt.rs` (2417 lines) NOT YET SPLIT
- Contains intertwined types: `FrequencyCounter`, `ClusterResult`, `Token`, `TokenBuffer`, `ProgressiveTokenBuffer`, `HuffmanOptimizer`
- Plan: Split into `huffman/optimize/{mod,frequency,cluster,tree,canonical}.rs`