Skip to main content

Crate rapidgzip_core

Crate rapidgzip_core 

Source
Expand description

Parallel decoding of gzip, zlib, and raw DEFLATE.

rapidgzip-core decodes single-member and concatenated gzip, BGZF, zlib, and raw DEFLATE. It follows rapidgzip’s marker/window algorithm for parallel decoding and uses zlib-rs as its inflate backend. Encoding is outside this crate’s current scope. Index construction, persistence, and decoded-output seeking are available through explicit opt-in APIs.

§Formats

Strict Format::Gzip is the default. DecoderBuilder::format selects zlib or raw DEFLATE explicitly; DecoderBuilder::auto_detect_format recognizes gzip or zlib without consuming their prefix. Raw DEFLATE has no identifying header and is never guessed.

Gzip checks every member’s CRC32 and ISIZE. Zlib validates CMF/FLG, enforces its declared history window, and checks Adler-32. Raw DEFLATE has no checksum, so success establishes structural validity and exact source consumption. DecoderBuilder::expected_uncompressed_size can require an exact decoded size for any format.

§Output interfaces

§Example

use rapidgzip_core::Decoder;
use std::io;

let decoder = Decoder::builder().decoder_threads(8).build()?;
let mut reader = decoder.open("reads.fastq.gz")?;
let control = reader.handle();
control.set_worker_limit(4)?;
io::copy(&mut reader, &mut io::sink())?;
let report = reader.finish()?;
assert!(report.member_count >= 1);

§Verification and errors

Reaching reader EOF or receiving a successful DecodeReport means the complete compressed input passed every check carried by its selected container. Dropping a DecoderReader before EOF cancels the unread work; call DecoderReader::finish when decoded bytes are no longer needed but complete validation is.

Decoding can emit a verified prefix before discovering later corruption or an I/O failure. Previously written or read bytes are not rolled back.

§Input and concurrency

Compressed input implements ReadAt, allowing bounded worker tasks to use positional reads without a shared cursor. Implementations are supplied for files on Unix and Windows, in-memory byte storage, std::sync::Arc, and Box. The source length and contents must remain stable during decoding.

§Non-seekable input

Decoder::decode_stream and Decoder::stream_reader accept any std::io::Read, so standard input, a FIFO, a process substitution, or a socket can be decoded. Decoder::open routes non-regular paths accepted by std::fs::File::open, such as FIFOs and character devices, to the same sequential engine.

Validation is identical: such a source runs the same sequential zlib-rs path that the parallel paths use as their authoritative fallback, sharing framing, trailer checks, trailing-data detection, and output bounds. It is not decoded in parallel, because every parallel path needs positional reads. Telemetry retains the builder’s configured worker budget while reporting an effective target of one and zero spawned decoder/auxiliary threads. Nothing is spooled: input memory is one DecoderBuilder::input_page_size window. DecoderReader advances the streaming inflater synchronously from std::io::Read::read, so dropping it immediately drops the source and cannot strand a thread blocked on input.

§Worker budgets and runtime control

DecoderBuilder::decoder_threads sets immutable per-operation headroom; it is neither an eager thread count nor a promise that every input will use that width. Parallel paths grow an elastic worker population from an affinity-, budget-, and workload-aware bootstrap, probe wider settings only when enough independent work exists, and prefer a narrower setting when its measured throughput is effectively tied. Sequential and short inputs can therefore use much less than the configured maximum.

A cloned DecoderHandle remains usable after a DecoderReader moves into another component. Its controls have deliberately different meanings:

Ignoring temporary consumer backpressure, the per-operation target is conceptually:

min(worker_limit, max(adaptive_target, requested_workers))

Available tasks and an optional shared-pool grant can reduce actual concurrency further. Excess workers finish their current task and retire; sustained reader backpressure also reduces local admission automatically. DecoderStats separates configured headroom, application policy, the adaptive target, pool-limited admission, executing tasks, and live threads. Its snapshots are instantaneous rather than high-water measurements.

§Shared decode budgets

DecoderPool is an opt-in process-wide execution budget for applications decoding several files at once. Clone one pool into each reusable DecoderBuilder::decoder_pool configuration. Stable max-min spawn allowances follow attached decoders’ declared demand, while CPU-intensive tasks acquire a fair global slot and release it before a bounded output handoff would block. Completion releases reservations, and explicit demand or pool-limit changes redistribute them immediately. Without a pool, the existing private elastic-worker behavior is unchanged.

use rapidgzip_core::{Decoder, DecoderPool};
let pool = DecoderPool::builder().workers(16).build()?;
let decoder = Decoder::builder()
    .decoder_threads(16)
    .decoder_pool(pool.clone())
    .build()?;
let reader = decoder.open("reads.fastq.gz")?;
let control = reader.handle();
control.request_workers(8)?;
pool.set_worker_limit(12)?;

DecoderHandle::request_workers sets a persistent growth floor distinct from the hard ceiling set by DecoderHandle::set_worker_limit. DecoderStats::desired_workers exposes demand before pool contention and DecoderStats::pool_limited identifies shared-budget pressure. Algorithm admission uses immutable configured capacity; a low live throttle therefore cannot permanently latch an otherwise suitable input to sequential decode.

§Structural analysis

Decoder::analyze and Decoder::analyze_stream verify the complete input while returning an Analysis of container streams, DEFLATE blocks, dynamic Huffman alphabets, symbol composition, and predecessor-window use. AnalyzeOptions bounds retained streams, blocks, optional gzip metadata, and individual back-references. Exact aggregate reference statistics remain available when detail retention is disabled or exhausted. The causal walk is single-threaded and keeps only one 32 KiB decoded history window.

§Random access

Decoder::decode_with_index and Decoder::reader_with_index collect a DeflateIndex only when requested, leaving DecodeReport small and Copy. The streaming counterparts collect a coarser member-boundary index while reading a forward-only source. The native format represents every supported container; GZIDX, htslib BGZF .gzi, and gztool are gzip-family formats and reject incompatible export.

Decoder::decode_from_index and Decoder::reader_from_index reuse an existing index for strict parallel full-stream decoding. Every worker must reach the next checkpoint’s exact compressed bit and decompressed byte offsets; invalid or source-mismatched indexes never trigger an ordinary fallback. The reader remains std::io::Read + Send and exposes the usual runtime worker controls. Concatenated and empty gzip members and BGZF .gzi indexes are supported without weakening whole-stream verification.

IndexedReader implements std::io::Read + std::io::Seek over a stable ReadAt source. Framing-start checkpoints permit complete gzip or zlib verification; an interior checkpoint cannot authenticate bytes skipped earlier because indexes do not store prefix checksum state. Raw DEFLATE has no checksum to authenticate.

§Line counting and seeking

DecoderBuilder::count_lines optionally counts newline bytes on final ordered output. The scalar result is returned in DecodeReport::line_count, so DecodeReport remains Copy. When the same operation explicitly builds an index, each retained checkpoint and the index total receive exact line metadata. IndexedReader::seek_to_line then seeks to a zero-based line without scanning from the source origin. Counting is disabled by default.

Re-exports§

pub use index::Checkpoint;
pub use index::CheckpointKind;
pub use index::DeflateIndex;
pub use index::IndexError;
pub use index::IndexKind;
pub use index::IndexOptions;
pub use index::IndexReadOptions;
pub use index::StoredWindow;
pub use index::WindowMap;
pub use index::WindowStorage;

Modules§

index
Random-access indexes for gzip, zlib, and raw-DEFLATE sources.
parallel
Building blocks for rapidgzip’s speculative marker/window algorithm.

Structs§

AlphabetShape
Shape of one alphabet declared by a dynamic-Huffman block.
Analysis
Complete deterministic structural analysis of an input.
AnalyzeOptions
Limits controlling the memory retained by structural analysis.
Backreference
One retained reference into the predecessor window.
BlockAnalysis
Structural facts for one DEFLATE block.
ConfigError
Invalid decoder configuration.
DecodeReport
Statistics produced after the complete stream has been verified.
Decoder
Immutable, reusable decompressor configuration.
DecoderBuilder
Builder for an immutable, reusable Decoder.
DecoderHandle
Cloneable telemetry and control handle for a running crate::DecoderReader.
DecoderPool
Cloneable process-wide budget shared by multiple decoders.
DecoderPoolConfigError
Invalid DecoderPool configuration.
DecoderPoolLimitError
Invalid runtime worker limit for a DecoderPool.
DecoderPoolStats
Approximate process-wide snapshot of a DecoderPool.
DecoderReader
Owned parallel decoder output implementing Read and Send.
DecoderStats
Approximate, lock-free snapshot of a running decoder.
GzipHeaderFields
Complete gzip header metadata for one member.
IndexedDecodeReport
Result of a verified decode that also collected a random-access index.
IndexedReader
A Read and Seek view of decompressed bytes described by an index.
IndexingDecoderReader
Owned decoded output that publishes a random-access index at verified EOF.
StreamAnalysis
One gzip member, zlib stream, or raw-DEFLATE stream.
WorkerLimitError
Invalid runtime decoder-worker limit.
WorkerRequestError
Invalid persistent decoder growth request.
ZlibHeaderFields
RFC 1950 header fields for one zlib stream.

Enums§

AnalysisCounter
A checked counter maintained by structural analysis.
AnalysisErrorKind
The reason structural analysis stopped without accepting the input.
AnalysisResource
An analysis-owned collection or byte budget.
BlockType
Encoding selected by one DEFLATE block.
DecodeError
A terminal decoding error.
DecoderPath
Decoder implementation selected for the current input.
DecoderPressure
Current high-level constraint on decoder progress.
DeflateErrorKind
The reason a DEFLATE stream was rejected.
Format
Container framing around a DEFLATE stream.
GzipErrorKind
The reason a gzip container was rejected.
IndexDecodeError
Failure while decoding through a caller-supplied index.
IndexedReaderError
Failure while opening an IndexedReader.
IndexingError
Failure of an operation that decodes and builds an index.
StreamFooter
Verified container trailer ending one analyzed stream.
StreamHeader
Container header beginning one analyzed stream.
ZlibErrorKind
The reason an RFC 1950 zlib container was rejected.

Traits§

ReadAt
Thread-safe positional compressed input.