why cant # ljar-rs Backlog
Working notes that survive token-limit truncation. Cross-check 2026-05-19.
---
## Cross-check: lbzip2-rs vs ljar-rs (2026-05-19)
DESIGN.md and `src/bin/ljar.rs` are now aligned. The binary implements the
full 5-thread pipeline mirroring `lbzip2-rs/src/bin/lbunzip2.rs`:
| Reader thread | dedicated | ✅ | ✅ |
| Workers | N dedicated threads on channel | ✅ | ✅ |
| Collector thread | dedicated | ✅ | ✅ |
| Writer thread | dedicated | ✅ | ✅ |
| Concurrent in-flight chunks | 6 (ring slots) | ✅ | ✅ |
| Workers idle between chunks | never | never | never |
| RING_SLOTS = 6 | 6 actual slots | ✅ | ✅ |
| No-flush fallback | single segment | N/A (bzip2 always has blocks) | ✅ accumulate + single segment |
### Bug fixed (2026-05-19)
`decode_segment_into` in `chunk.rs` was passing `out_pos = 0` to
`miniz_oxide::decompress` on every iteration. With
`TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF`, the decompressor needs to see all
prior output for LZ77 back-references. Fixed to pass the cumulative output
length as `out_pos`.
---
## Backlog: full pipeline rewrite
### B1 — ✅ DONE: Full 5-thread pipeline
`src/bin/ljar.rs` now mirrors `lbzip2-rs/src/bin/lbunzip2.rs` exactly:
Reader → Main(carry+split) → N Workers → Collector → Writer, with 6
cycling ring slots and `Arc<Mutex<Receiver<WorkItem>>>`.
Small entries (< 4 MB) bypass ring slots as ad-hoc single-segment chunks.
Entries without Z_FULL_FLUSH boundaries accumulate into a fallback buffer
and decompress as one segment (single-core).
### B2 — ✅ DONE: Entry-boundary handling
Implemented via `FilledSlot` with `is_first_of_entry` / `is_last_of_entry` flags.
Reader signals boundaries; main thread resets carry at entry transitions.
### B3 — ✅ DONE: Slot-pool sizing
6 slots = up to 1.2 GB in flight. Entries larger than that still work: reader
blocks on `free_rx.recv()` until a slot is recycled, then fills the next chunk.
Pipeline depth is bounded but throughput is the same. Documented in constants.
### B4 — ✅ DONE: Fallback for entries without Z_FULL_FLUSH
Entries without flush boundaries (standard `jar` tool output) now decompress
correctly. Small entries (< 4 MB) are single-segment chunks through the normal
pipeline. Multi-chunk entries without flush boundaries accumulate into a fallback
buffer and decompress as a single segment when the entry ends.
### B5 — Benchmark harness
Once B1 lands, benchmark against:
- `unzip` (single-core C reference)
- `bsdtar -xf` (uses libarchive)
- `python -m zipfile -e` (single-core)
- ljar-rs v0 (current serial binary)
- ljar-rs v1 (full pipeline)
Test JARs:
- Synthetic: 1 × 1 GB DEFLATE entry with Z_FULL_FLUSH every 32 MB
- Real: large Spring Boot fat-jar (~100 MB, many small entries)
- Real: any large Maven artifact downloaded into /home/rickard/work
### B6 — Zero-copy decompressed output (DESIGN Rule 5.2 + 6.3)
Currently `decode_segment_into` allocates a `Vec<u8>` per segment. The lbzip2 pattern
`reserve + set_len + decompress_into` is in place but the segment Vec itself is
freshly allocated each call. To match Rule 6.3 (zero-copy offset variant):
- Pre-allocate one `Vec<u8>` of `uncompressed_size` per entry
- Workers write directly into `output[segment_offset..]`
- Writer thread receives `(entry_idx, written_len)` only — no copy
This is the production zero-copy path. Convenience API (`decompress_jar`) keeps
the current allocating behaviour.
### B7 — Custom DEFLATE decompressor (`src/inflate/`)
Cherry-picks the best from libdeflate, zlib-ng, zlib-rs, and lbzip2-rs:
**Architecture:** libdeflate's full-buffer model (no circular window — ljar
already has complete segments in memory). Eliminates all window arithmetic,
wrap-around masking, and "copy from window" paths.
**Key design choices:**
- 11-bit litlen first-level Huffman table (libdeflate) — eliminates subtable
lookups for virtually all real streams. Packed u32 entries with length base +
extra bits baked in — one expression per match decode.
- Branchless 64-bit bit refill (zlib-ng XOR trick) — 4 instructions, no branch.
- 3-literal unroll per iteration (zlib-ng) — best for literal-heavy class files.
- Preload-before-copy pattern (libdeflate) — next Huffman entry loaded BEFORE
match copy runs, hiding L1 load latency behind copy instructions.
- Const-generic `<N>` SIMD match copy (zlib-rs) — LLVM emits `vmovdqu ymm` for
N=32 (AVX2), `movdqu xmm` for N=16 (SSE2/NEON). Clean Rust, no intrinsics.
- AVX2 `_mm256_shuffle_epi8` tiling for short back-refs dist 2..7 (zlib-ng) —
single instruction tiles pattern across 32-byte register.
- Thread-local table pool (lbzip2-rs pattern) — zero heap allocation per block.
- `inflate_into(&mut [u8])` zero-copy API — caller owns output buffer.
- `#[cold] #[inline(never)]` slow paths (lbzip2-rs) — keep fast path hot.
**Cache budget:** ~13 KB total hot working set (litlen 9.4 KB + dist 1.6 KB +
precode 0.5 KB + BitReader 32 B + scratch 1.3 KB) — fits entirely in 32 KB L1-D.
**Implementation order:**
1. `bitreader.rs` — branchless refill, peek/consume, BMI2 variant
2. `tables.rs` — packed u32 entry format, 11-bit litlen, table build
3. `fixed.rs` — pre-built fixed Huffman tables (static)
4. `copy.rs` — SIMD match copy with const-generic N, AVX2 short-distance
5. `fastloop.rs` — the hot loop with preload-before-copy
6. `mod.rs` — public API, block parsing, dynamic header decode
7. Integration tests: verify against miniz_oxide on all 20 Maven JARs
8. Benchmarks: compare against miniz_oxide, measure throughput per core
Full design in `src/inflate/DESIGN.md`.
---
## Where things live
| Full-flush scanner | `src/deflate_scan.rs` |
| Chunk split + parallel decode | `src/chunk.rs` |
| Central directory parsing | `src/central_dir.rs` |
| Per-entry decompress (library API) | `src/entry.rs` |
| Streaming iterator | `src/reader.rs` |
| In-memory parallel batch | `src/parallel.rs` |
| Public API entry points | `src/lib.rs` |
| **CLI binary (v1 full pipeline)** | `src/bin/ljar.rs` |
Reference impl:
- `/run/media/rickard/T9/git/lbzip2-rs/src/bin/lbunzip2.rs` — the model for B1
- `/run/media/rickard/T9/git/lbzip2-rs/src/chunk.rs` — split_chunk pattern
---
## Useful test scratch
Generate test JARs in `/home/rickard/work/` (per user preference):
```bash
python3 -c "
import zipfile, io
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w', zipfile.ZIP_DEFLATED) as zf:
zf.writestr('Hello.class', b'cafebabe' * 1000)
open('/home/rickard/work/test.jar','wb').write(buf.getvalue())
"
```
Run:
```bash
cargo run --bin ljar -- /home/rickard/work/test.jar # list
cargo run --bin ljar -- /home/rickard/work/test.jar /home/rickard/work/out # extract
```