Crate rapidgzip_core

Expand description

Parallel decoding of gzip, zlib, and raw DEFLATE.

rapidgzip-core decodes single-member and concatenated gzip, BGZF, zlib, and raw DEFLATE. It follows rapidgzip’s marker/window algorithm for parallel decoding and uses zlib-rs as its inflate backend. Encoding is outside this crate’s current scope. Index construction, persistence, and decoded-output seeking are available through explicit opt-in APIs.

§Formats

Strict Format::Gzip is the default. DecoderBuilder::format selects zlib or raw DEFLATE explicitly; DecoderBuilder::auto_detect_format recognizes gzip or zlib without consuming their prefix. Raw DEFLATE has no identifying header and is never guessed.

Gzip checks every member’s CRC32 and ISIZE. Zlib validates CMF/FLG, enforces its declared history window, and checks Adler-32. Raw DEFLATE has no checksum, so success establishes structural validity and exact source consumption. DecoderBuilder::expected_uncompressed_size can require an exact decoded size for any format.

§Output interfaces

Decoder::decode is the lower-overhead push interface, and Decoder::decode_path adds automatic regular/non-regular path routing. Both write on the calling thread, so std::io::Write need not be Send.
Decoder::reader and Decoder::open return an owned DecoderReader implementing std::io::Read + Send. This is suitable for parsers that take Box<dyn Read + Send>, including paraseq.
Decoder::decode_stream and Decoder::stream_reader are the same two interfaces for non-seekable input; see below.

§Example

use rapidgzip_core::Decoder;
use std::io;

let decoder = Decoder::builder().decoder_threads(8).build()?;
let mut reader = decoder.open("reads.fastq.gz")?;
let control = reader.handle();
control.set_worker_limit(4)?;
io::copy(&mut reader, &mut io::sink())?;
let report = reader.finish()?;
assert!(report.member_count >= 1);

§Verification and errors

Reaching reader EOF or receiving a successful DecodeReport means the complete compressed input passed every check carried by its selected container. Dropping a DecoderReader before EOF cancels the unread work; call DecoderReader::finish when decoded bytes are no longer needed but complete validation is.

Decoding can emit a verified prefix before discovering later corruption or an I/O failure. Previously written or read bytes are not rolled back.

§Input and concurrency

Compressed input implements ReadAt, allowing bounded worker tasks to use positional reads without a shared cursor. Implementations are supplied for files on Unix and Windows, in-memory byte storage, std::sync::Arc, and Box. The source length and contents must remain stable during decoding.

§Non-seekable input

Decoder::decode_stream and Decoder::stream_reader accept any std::io::Read, so standard input, a FIFO, a process substitution, or a socket can be decoded. Decoder::open routes non-regular paths accepted by std::fs::File::open, such as FIFOs and character devices, to the same sequential engine.

Validation is identical: such a source runs the same sequential zlib-rs path that the parallel paths use as their authoritative fallback, sharing framing, trailer checks, trailing-data detection, and output bounds. It is not decoded in parallel, because every parallel path needs positional reads. Telemetry retains the builder’s configured worker budget while reporting an effective target of one and zero spawned decoder/auxiliary threads. Nothing is spooled: input memory is one DecoderBuilder::input_page_size window. DecoderReader advances the streaming inflater synchronously from std::io::Read::read, so dropping it immediately drops the source and cannot strand a thread blocked on input.

DecoderBuilder::decoder_threads sets a maximum worker budget rather than eagerly creating that many threads. Parallel paths grow an elastic worker population from an affinity- and budget-aware bootstrap. A cloned DecoderHandle provides lock-free telemetry and can change the runtime ceiling after a DecoderReader moves into another component. Excess workers finish their current task and retire; sustained reader backpressure also reduces admission automatically.

§Structural analysis

Decoder::analyze and Decoder::analyze_stream verify the complete input while returning an Analysis of container streams, DEFLATE blocks, dynamic Huffman alphabets, symbol composition, and predecessor-window use. AnalyzeOptions bounds retained streams, blocks, optional gzip metadata, and individual back-references. Exact aggregate reference statistics remain available when detail retention is disabled or exhausted. The causal walk is single-threaded and keeps only one 32 KiB decoded history window.

§Random access

Decoder::decode_with_index and Decoder::reader_with_index collect a DeflateIndex only when requested, leaving DecodeReport small and Copy. The streaming counterparts collect a coarser member-boundary index while reading a forward-only source. The native format represents every supported container; GZIDX, htslib BGZF .gzi, and gztool are gzip-family formats and reject incompatible export.

Decoder::decode_from_index and Decoder::reader_from_index reuse an existing index for strict parallel full-stream decoding. Every worker must reach the next checkpoint’s exact compressed bit and decompressed byte offsets; invalid or source-mismatched indexes never trigger an ordinary fallback. The reader remains std::io::Read + Send and exposes the usual runtime worker controls. Concatenated and empty gzip members and BGZF .gzi indexes are supported without weakening whole-stream verification.

IndexedReader implements std::io::Read + std::io::Seek over a stable ReadAt source. Framing-start checkpoints permit complete gzip or zlib verification; an interior checkpoint cannot authenticate bytes skipped earlier because indexes do not store prefix checksum state. Raw DEFLATE has no checksum to authenticate.

§Line counting and seeking

DecoderBuilder::count_lines optionally counts newline bytes on final ordered output. The scalar result is returned in DecodeReport::line_count, so DecodeReport remains Copy. When the same operation explicitly builds an index, each retained checkpoint and the index total receive exact line metadata. IndexedReader::seek_to_line then seeks to a zero-based line without scanning from the source origin. Counting is disabled by default.

Re-exports§

pub use index::Checkpoint;
pub use index::CheckpointKind;
pub use index::DeflateIndex;
pub use index::IndexError;
pub use index::IndexKind;
pub use index::IndexOptions;
pub use index::IndexReadOptions;
pub use index::StoredWindow;
pub use index::WindowMap;
pub use index::WindowStorage;

Modules§

index: Random-access indexes for gzip, zlib, and raw-DEFLATE sources.
parallel: Building blocks for rapidgzip’s speculative marker/window algorithm.

Structs§

AlphabetShape: Shape of one alphabet declared by a dynamic-Huffman block.
Analysis: Complete deterministic structural analysis of an input.
AnalyzeOptions: Limits controlling the memory retained by structural analysis.
Backreference: One retained reference into the predecessor window.
BlockAnalysis: Structural facts for one DEFLATE block.
ConfigError: Invalid decoder configuration.
DecodeReport: Statistics produced after the complete stream has been verified.
Decoder: Immutable, reusable decompressor configuration.
DecoderBuilder: Builder for an immutable, reusable Decoder.
DecoderHandle: Cloneable telemetry and control handle for a running crate::DecoderReader.
DecoderReader: Owned parallel decoder output implementing Read and Send.
DecoderStats: Approximate, lock-free snapshot of a running decoder.
GzipHeaderFields: Complete gzip header metadata for one member.
IndexedDecodeReport: Result of a verified decode that also collected a random-access index.
IndexedReader: A Read and Seek view of decompressed bytes described by an index.
IndexingDecoderReader: Owned decoded output that publishes a random-access index at verified EOF.
StreamAnalysis: One gzip member, zlib stream, or raw-DEFLATE stream.
WorkerLimitError: Invalid runtime decoder-worker limit.
ZlibHeaderFields: RFC 1950 header fields for one zlib stream.

Enums§

AnalysisCounter: A checked counter maintained by structural analysis.
AnalysisErrorKind: The reason structural analysis stopped without accepting the input.
AnalysisResource: An analysis-owned collection or byte budget.
BlockType: Encoding selected by one DEFLATE block.
DecodeError: A terminal decoding error.
DecoderPath: Decoder implementation selected for the current input.
DecoderPressure: Current high-level constraint on decoder progress.
DeflateErrorKind: The reason a DEFLATE stream was rejected.
Format: Container framing around a DEFLATE stream.
GzipErrorKind: The reason a gzip container was rejected.
IndexDecodeError: Failure while decoding through a caller-supplied index.
IndexedReaderError: Failure while opening an IndexedReader.
IndexingError: Failure of an operation that decodes and builds an index.
StreamFooter: Verified container trailer ending one analyzed stream.
StreamHeader: Container header beginning one analyzed stream.
ZlibErrorKind: The reason an RFC 1950 zlib container was rejected.

Traits§

ReadAt: Thread-safe positional compressed input.