Module compression

Data compression module

Provides utilities for compressing and decompressing scientific data:

• Lossless compression algorithms (GZIP, ZSTD, LZ4, BZIP2)
• Array compression with metadata preservation
• Chunked compression for large datasets
• Compression level configuration Compression utilities for scientific data

This module provides functionality for compressing and decompressing data using various algorithms suitable for scientific computing. It focuses on lossless compression to ensure data integrity while reducing storage requirements.

Features

• Multiple compression algorithms (GZIP, ZSTD, LZ4, BZIP2)
• Configurable compression levels
• Memory-efficient compression of large datasets
• Metadata preservation during compression
• Array-specific compression optimizations

Sub-modules

• ndarray: Specialized compression utilities for ndarray types

Examples

use scirs2_io::compression::{compress_data, decompress_data, CompressionAlgorithm};
use std::fs::File;
use std::io::prelude::*;

// Compress some data using ZSTD with default compression level
let data = b"Large scientific dataset with repetitive patterns";
let compressed = compress_data(data, CompressionAlgorithm::Zstd, None).unwrap();

// Save the compressed data to a file
let mut file = File::create("data.zst").unwrap();
file.write_all(&compressed).unwrap();

// Later, read and decompress the data
let mut compressed_data = Vec::new();
File::open("data.zst").unwrap().read_to_end(&mut compressed_data).unwrap();
let original = decompress_data(&compressed_data, CompressionAlgorithm::Zstd).unwrap();
assert_eq!(original, data);

Modules

ndarray

Compression utilities for ndarray types

Structs

CompressionBenchmarkResult

Results from compression benchmarking

CompressionInfo

Information about a compression algorithm

FileCompressionInfo

Information about a file’s compression status

ParallelCompressionConfig

Configuration for parallel compression/decompression operations

ParallelCompressionStats

Statistics for parallel compression/decompression operations

TransparentFileHandler

Transparent file handler that automatically handles compression/decompression

Enums

CompressionAlgorithm

Compression algorithms supported by the library

Functions

algorithm_info

Get information about a specific compression algorithm

benchmark_compression_algorithms

Benchmark compression performance for different algorithms and configurations

compress_data

Compress data using the specified algorithm and compression level

compress_data_parallel

Compress data in parallel using multiple threads

compress_file

Compress a file using the specified algorithm and save it to a new file

compress_file_parallel

Compress a file in parallel and save it to a new file

compression_ratio

Calculate the compression ratio for the given data and algorithm

copy_file_transparent

Convenient function to copy a file with transparent compression/decompression using global handler

decompress_data

Decompress data using the specified algorithm

decompress_data_parallel

Decompress data in parallel using multiple threads

decompress_file

Decompress a file using the specified algorithm and save it to a new file

decompress_file_parallel

Decompress a file in parallel and save it to a new file

detect_compression_from_bytes

Detect compression algorithm from magic bytes

file_info_transparent

Convenient function to get file compression info using global handler

global_handler

Get a reference to the global transparent file handler

init_global_handler

Initialize the global transparent file handler

read_file_transparent

Convenient function to read a file with automatic decompression using global handler

write_file_transparent

Convenient function to write a file with automatic compression using global handler