Module adaptive_memory_compression 

Adaptive Memory Compression for Advanced-Large Sparse Matrices

This module provides advanced memory management and compression techniques specifically designed for handling advanced-large sparse matrices that exceed available system memory.

Architecture

The adaptive memory compression system consists of several interconnected components:

• Configuration: Flexible configuration system for different compression strategies
• Cache Management: Intelligent block caching with LRU eviction and access tracking
• Access Tracking: Pattern analysis for optimizing compression and caching decisions
• Compression: Multiple compression algorithms optimized for sparse data
• Out-of-Core Storage: Seamless handling of matrices larger than available memory
• Memory Mapping: Efficient file-based storage with memory mapping support
• Statistics: Comprehensive performance tracking and optimization guidance

Usage

use scirs2_sparse::adaptive_memory_compression::{
    AdaptiveMemoryCompressor, AdaptiveCompressionConfig, CompressionAlgorithm
};

// Create configuration
let config = AdaptiveCompressionConfig::new()
    .with_memory_budget(8 * 1024 * 1024 * 1024) // 8GB
    .with_compression_algorithm(CompressionAlgorithm::Adaptive)
    .with_out_of_core(true);

// Create compressor
let mut compressor = AdaptiveMemoryCompressor::new(config)?;

// Compress a sparse matrix
let compressed = compressor.compress_matrix(
    1,          // matrix_id
    1000,       // rows
    &indptr,    // CSR indptr
    &indices,   // CSR indices
    &data       // CSR data
)?;

Performance Optimization

The system automatically learns from access patterns and adapts compression strategies:

// Get performance statistics
let stats = compressor.get_stats();
println!("Compression ratio: {:.2}", stats.compression_ratio);
println!("Cache hit rate: {:.2}%", stats.cache_hit_ratio * 100.0);

// Manual optimization based on access patterns
compressor.optimize_for_sequential_access();
compressor.optimize_for_random_access();

Re-exports

pub use cache::BlockId;

pub use compressed_data::BlockType;

pub use compressed_data::CompressedBlock;

pub use compressed_data::CompressedMatrix;

pub use compressor::AdaptiveMemoryCompressor;

pub use config::AdaptiveCompressionConfig;

pub use config::CompressionAlgorithm;

pub use stats::CompressionMetadata;

pub use stats::CompressionStats;

pub use stats::MemoryStats;

pub use access_tracking::AccessType;

pub use compression::CompressionEngine;

pub use compression::CompressionResult;

pub use memory_mapping::MemoryMappedFile;

pub use out_of_core::OutOfCoreManager;

pub use stats::AccessPatternType;

Modules

access_tracking

Access pattern tracking for adaptive memory compression

cache

Block cache management for adaptive memory compression

compressed_data

Compressed data structures for adaptive memory compression

compression

Compression algorithms and engine for adaptive memory compression

compressor

Adaptive Memory Compressor

config

Configuration for adaptive memory compression

memory_mapping

Memory-Mapped File Operations

out_of_core

Out-of-Core Memory Management

stats

Statistics and metadata tracking for adaptive memory compression