Crate s4_codec 

S4 圧縮 codec layer。バックエンドを差し替え可能にする中立 trait を提供する。

採用 backend (2026-05 検討)

• nvCOMP (NVIDIA proprietary、要 license 確認): Bitcomp / gANS / zstd-GPU
• DietGPU (Meta, MIT): ANS-only、license clean な fallback
• CPU zstd: GPU 無し環境向け究極の fallback / test bed

Re-exports

pub use registry::CodecRegistry;

pub use dispatcher::CodecDispatcher;

Modules

cpu_gzip

RFC 1952 gzip codec via flate2 (v0.4 #26).

cpu_zstd

CPU zstd backend — GPU 非搭載環境向け究極の fallback、および test bed。

dietgpu

DietGPU (Meta, MIT) backend ラッパー — nvCOMP ライセンス障害時の OSS fallback。

dispatcher

PUT 時にどの codec で圧縮するかを選ぶ dispatcher。

index

Frame index — Range GET の partial fetch を可能にするための sidecar object 形式。

multipart

Multipart upload で使う on-the-wire フレーム形式。

nvcomp

nvCOMP (NVIDIA proprietary) backend ラッパー。

passthrough

無圧縮 codec — テストおよび圧縮無効化フラグ用。

registry

複数 Codec を保持し、CodecKind ベースで dispatch するレジストリ。

Structs

ChunkManifest

圧縮済 chunk のメタ情報。S3 オブジェクトの metadata に格納される。

CompressTelemetry

v0.8 #55: per-op telemetry returned by CodecRegistry::compress_with_telemetry / decompress_with_telemetry. Lets the s4-server caller stamp Prometheus metrics (s4_gpu_compress_seconds, s4_gpu_throughput_bytes_per_sec, s4_gpu_oom_total) without s4-codec needing a metrics dep itself — callback pattern keeps the codec dep tree slim.

ParseCodecKindError

Enums

CodecError

codec 操作のエラー型。anyhow::Error ではなく専用型にすることで、上位 (S4Service) が HTTP エラーコードを意味的に出し分けやすくする。

CodecKind

圧縮 codec の種類 (manifest に記録、後段の decompress で codec を確定するために使う)

Constants

DECOMPRESS_BOOTSTRAP_CAPACITY

v0.8.6 #89: bootstrap capacity for the decompressed-output Vec so the Vec::with_capacity(original_size) pre-allocation can no longer be driven into RSS-OOM by a forged manifest. Small enough (1 MiB) that even an attacker claiming original_size = u32::MAX only reserves 1 MiB up front; read_to_end grows the buffer as actual decompressed bytes arrive (capped at manifest.original_size + 1024 by the existing decompression-bomb guard).

MAX_DECOMPRESSED_BYTES

v0.8.6 #89: maximum decompressed payload size honoured at decompress entry by every codec. Manifests claiming a larger original_size are rejected pre-allocation as forged / corrupted, so a malicious manifest cannot drive Vec::with_capacity(huge) into an OOM (memory-DoS) before the CRC check ever runs.

Traits

Codec

pluggable な圧縮 backend trait。

Functions

looks_like_oom

v0.8 #55: heuristic OOM classifier. nvCOMP / cudarc surface OOM as a CodecError::Backend(anyhow!("...out of memory...")) (the underlying CUDA driver returns CUDA_ERROR_OUT_OF_MEMORY which cudarc / nvCOMP stringify); we substring-match for the well-known fragments so the metric stamp doesn’t need to thread a typed error variant through the FFI boundary. Returns true only on a high-confidence match; non-OOM backend errors (CRC mismatch, IO error, etc.) yield false and are stamped as plain s4_requests_total{result="err"} without bumping the OOM counter.

validate_decompress_manifest

v0.8.6 #89: shared pre-allocation manifest validator invoked by every decompress path (CpuZstd / CpuGzip / nvCOMP Zstd / Bitcomp / GDeflate). Centralising the check keeps every decompress site using identical limits and error shapes, so one missed update can’t reintroduce the alloc-before-validate bug. Returns the usize-narrowed original_size ready for Vec::with_capacity, or a typed CodecError the caller propagates verbatim.