Expand description
Parallel decoding of gzip, zlib, and raw DEFLATE.
rapidgzip-core decodes single-member and concatenated gzip, BGZF, zlib,
and raw DEFLATE. It follows rapidgzip’s marker/window algorithm for parallel
decoding and uses zlib-rs as its inflate backend. Encoding is outside this
crate’s current scope. Index construction, persistence, and decoded-output
seeking are available through explicit opt-in APIs.
§Formats
Strict Format::Gzip is the default. DecoderBuilder::format selects
zlib or raw DEFLATE explicitly; DecoderBuilder::auto_detect_format
recognizes gzip or zlib without consuming their prefix. Raw DEFLATE has no
identifying header and is never guessed.
Gzip checks every member’s CRC32 and ISIZE. Zlib validates CMF/FLG, enforces
its declared history window, and checks Adler-32. Raw DEFLATE has no
checksum, so success establishes structural validity and exact source
consumption. DecoderBuilder::expected_uncompressed_size can require an
exact decoded size for any format.
§Output interfaces
Decoder::decodeis the lower-overhead push interface, andDecoder::decode_pathadds automatic regular/non-regular path routing. Both write on the calling thread, sostd::io::Writeneed not beSend.Decoder::readerandDecoder::openreturn an ownedDecoderReaderimplementingstd::io::Read+Send. This is suitable for parsers that takeBox<dyn Read + Send>, includingparaseq.Decoder::decode_streamandDecoder::stream_readerare the same two interfaces for non-seekable input; see below.
§Example
use rapidgzip_core::Decoder;
use std::io;
let decoder = Decoder::builder().decoder_threads(8).build()?;
let mut reader = decoder.open("reads.fastq.gz")?;
let control = reader.handle();
control.set_worker_limit(4)?;
io::copy(&mut reader, &mut io::sink())?;
let report = reader.finish()?;
assert!(report.member_count >= 1);§Verification and errors
Reaching reader EOF or receiving a successful DecodeReport means the
complete compressed input passed every check carried by its selected
container. Dropping a DecoderReader before EOF cancels the unread work;
call DecoderReader::finish when decoded bytes are no longer needed but
complete validation is.
Decoding can emit a verified prefix before discovering later corruption or an I/O failure. Previously written or read bytes are not rolled back.
§Input and concurrency
Compressed input implements ReadAt, allowing bounded worker tasks to use
positional reads without a shared cursor. Implementations are supplied for
files on Unix and Windows, in-memory byte storage, std::sync::Arc, and
Box. The source length and contents must remain stable during decoding.
§Non-seekable input
Decoder::decode_stream and Decoder::stream_reader accept any
std::io::Read, so standard input, a FIFO, a process substitution, or a
socket can be decoded. Decoder::open routes non-regular paths accepted by
std::fs::File::open, such as FIFOs and character devices, to the same
sequential engine.
Validation is identical: such a source runs the same sequential zlib-rs
path that the parallel paths use as their authoritative fallback, sharing
framing, trailer checks, trailing-data detection, and output bounds. It is
not decoded in parallel, because every parallel path needs positional reads.
Telemetry retains the builder’s configured worker budget while
reporting an effective target of one and zero spawned decoder/auxiliary
threads. Nothing is spooled: input memory is one
DecoderBuilder::input_page_size window. DecoderReader advances the
streaming inflater synchronously from std::io::Read::read, so dropping it
immediately drops the source and cannot strand a thread blocked on input.
§Worker budgets and runtime control
DecoderBuilder::decoder_threads sets immutable per-operation headroom;
it is neither an eager thread count nor a promise that every input will use
that width. Parallel paths grow an elastic worker population from an
affinity-, budget-, and workload-aware bootstrap, probe wider settings only
when enough independent work exists, and prefer a narrower setting when its
measured throughput is effectively tied. Sequential and short inputs can
therefore use much less than the configured maximum.
A cloned DecoderHandle remains usable after a DecoderReader moves
into another component. Its controls have deliberately different meanings:
DecoderHandle::set_worker_limitchanges a hard per-operation ceiling;DecoderHandle::request_workerssupplies a persistent growth floor beneath that ceiling; and- when configured,
DecoderPool::set_worker_limitchanges the aggregate execution ceiling shared by attached operations.
Ignoring temporary consumer backpressure, the per-operation target is conceptually:
min(worker_limit, max(adaptive_target, requested_workers))Available tasks and an optional shared-pool grant can reduce actual
concurrency further. Excess workers finish their current task and retire;
sustained reader backpressure also reduces local admission automatically.
DecoderStats separates configured headroom, application policy, the
adaptive target, pool-limited admission, executing tasks, and live threads.
Its snapshots are instantaneous rather than high-water measurements.
§Shared decode budgets
DecoderPool is an opt-in process-wide execution budget for applications
decoding several files at once. Clone one pool into each reusable
DecoderBuilder::decoder_pool configuration. Stable max-min spawn
allowances follow attached decoders’ declared demand, while CPU-intensive
tasks acquire a fair global slot and release it before a bounded output
handoff would block. Completion releases reservations, and explicit demand
or pool-limit changes redistribute them immediately. Without a pool, the
existing private elastic-worker behavior is unchanged.
use rapidgzip_core::{Decoder, DecoderPool};
let pool = DecoderPool::builder().workers(16).build()?;
let decoder = Decoder::builder()
.decoder_threads(16)
.decoder_pool(pool.clone())
.build()?;
let reader = decoder.open("reads.fastq.gz")?;
let control = reader.handle();
control.request_workers(8)?;
pool.set_worker_limit(12)?;DecoderHandle::request_workers sets a persistent growth floor distinct
from the hard ceiling set by DecoderHandle::set_worker_limit.
DecoderStats::desired_workers exposes demand before pool contention and
DecoderStats::pool_limited identifies shared-budget pressure. Algorithm
admission uses immutable configured capacity; a low live throttle therefore
cannot permanently latch an otherwise suitable input to sequential decode.
§Structural analysis
Decoder::analyze and Decoder::analyze_stream verify the complete
input while returning an Analysis of container streams, DEFLATE blocks,
dynamic Huffman alphabets, symbol composition, and predecessor-window use.
AnalyzeOptions bounds retained streams, blocks, optional gzip metadata,
and individual back-references. Exact aggregate reference statistics remain
available when detail retention is disabled or exhausted. The causal walk
is single-threaded and keeps only one 32 KiB decoded history window.
§Random access
Decoder::decode_with_index and Decoder::reader_with_index collect a
DeflateIndex only when requested, leaving DecodeReport small and
Copy. The streaming counterparts collect a coarser member-boundary index
while reading a forward-only source. The native format represents every
supported container; GZIDX, htslib BGZF .gzi, and gztool are gzip-family
formats and reject incompatible export.
Decoder::decode_from_index and Decoder::reader_from_index reuse an
existing index for strict parallel full-stream decoding. Every worker must
reach the next checkpoint’s exact compressed bit and decompressed byte
offsets; invalid or source-mismatched indexes never trigger an ordinary
fallback. The reader remains std::io::Read + Send and exposes the
usual runtime worker controls. Concatenated and empty gzip members and BGZF
.gzi indexes are supported without weakening whole-stream verification.
IndexedReader implements std::io::Read + std::io::Seek over a
stable ReadAt source. Framing-start checkpoints permit complete gzip or
zlib verification; an interior checkpoint cannot authenticate bytes skipped
earlier because indexes do not store prefix checksum state. Raw DEFLATE has
no checksum to authenticate.
§Line counting and seeking
DecoderBuilder::count_lines optionally counts newline bytes on final
ordered output. The scalar result is returned in
DecodeReport::line_count, so DecodeReport remains Copy. When the
same operation explicitly builds an index, each retained checkpoint and the
index total receive exact line metadata. IndexedReader::seek_to_line
then seeks to a zero-based line without scanning from the source origin.
Counting is disabled by default.
Re-exports§
pub use index::Checkpoint;pub use index::CheckpointKind;pub use index::DeflateIndex;pub use index::IndexError;pub use index::IndexKind;pub use index::IndexOptions;pub use index::IndexReadOptions;pub use index::StoredWindow;pub use index::WindowMap;pub use index::WindowStorage;
Modules§
- index
- Random-access indexes for gzip, zlib, and raw-DEFLATE sources.
- parallel
- Building blocks for rapidgzip’s speculative marker/window algorithm.
Structs§
- Alphabet
Shape - Shape of one alphabet declared by a dynamic-Huffman block.
- Analysis
- Complete deterministic structural analysis of an input.
- Analyze
Options - Limits controlling the memory retained by structural analysis.
- Backreference
- One retained reference into the predecessor window.
- Block
Analysis - Structural facts for one DEFLATE block.
- Config
Error - Invalid decoder configuration.
- Decode
Report - Statistics produced after the complete stream has been verified.
- Decoder
- Immutable, reusable decompressor configuration.
- Decoder
Builder - Builder for an immutable, reusable
Decoder. - Decoder
Handle - Cloneable telemetry and control handle for a running
crate::DecoderReader. - Decoder
Pool - Cloneable process-wide budget shared by multiple decoders.
- Decoder
Pool Config Error - Invalid
DecoderPoolconfiguration. - Decoder
Pool Limit Error - Invalid runtime worker limit for a
DecoderPool. - Decoder
Pool Stats - Approximate process-wide snapshot of a
DecoderPool. - Decoder
Reader - Owned parallel decoder output implementing
ReadandSend. - Decoder
Stats - Approximate, lock-free snapshot of a running decoder.
- Gzip
Header Fields - Complete gzip header metadata for one member.
- Indexed
Decode Report - Result of a verified decode that also collected a random-access index.
- Indexed
Reader - A
ReadandSeekview of decompressed bytes described by an index. - Indexing
Decoder Reader - Owned decoded output that publishes a random-access index at verified EOF.
- Stream
Analysis - One gzip member, zlib stream, or raw-DEFLATE stream.
- Worker
Limit Error - Invalid runtime decoder-worker limit.
- Worker
Request Error - Invalid persistent decoder growth request.
- Zlib
Header Fields - RFC 1950 header fields for one zlib stream.
Enums§
- Analysis
Counter - A checked counter maintained by structural analysis.
- Analysis
Error Kind - The reason structural analysis stopped without accepting the input.
- Analysis
Resource - An analysis-owned collection or byte budget.
- Block
Type - Encoding selected by one DEFLATE block.
- Decode
Error - A terminal decoding error.
- Decoder
Path - Decoder implementation selected for the current input.
- Decoder
Pressure - Current high-level constraint on decoder progress.
- Deflate
Error Kind - The reason a DEFLATE stream was rejected.
- Format
- Container framing around a DEFLATE stream.
- Gzip
Error Kind - The reason a gzip container was rejected.
- Index
Decode Error - Failure while decoding through a caller-supplied index.
- Indexed
Reader Error - Failure while opening an
IndexedReader. - Indexing
Error - Failure of an operation that decodes and builds an index.
- Stream
Footer - Verified container trailer ending one analyzed stream.
- Stream
Header - Container header beginning one analyzed stream.
- Zlib
Error Kind - The reason an RFC 1950 zlib container was rejected.
Traits§
- ReadAt
- Thread-safe positional compressed input.