Struct CompressedMemMappedArray 

pub struct CompressedMemMappedArray<A: Clone + Copy + 'static + Send + Sync> { /* private fields */ }

A memory-mapped array with transparent compression.

This struct provides a view into a compressed array stored on disk, with transparent decompression of blocks as they are accessed.

Implementations

impl<A: Clone + Copy + 'static + Send + Sync> CompressedMemMappedArray<A>

pub fn open<P: AsRef<Path>>(path: P) -> CoreResult<Self>

Open a compressed memory-mapped array from a file.

Arguments

• path - The path to the compressed file

Returns

A compressed memory-mapped array

pub fn open_with_cache<P: AsRef<Path>>( path: P, cache_size: usize, cache_ttl: Option<Duration>, ) -> CoreResult<Self>

Open a compressed memory-mapped array with custom cache settings.

Arguments

• path - The path to the compressed file
• cache_size - The maximum number of blocks to cache
• cache_ttl - The time-to-live for cached blocks

Returns

A compressed memory-mapped array

pub fn shape(&self) -> &[usize]

Get the shape of the array.

pub fn size(&self) -> usize

Get the total number of elements in the array.

pub fn ndim(&self) -> usize

Get the number of dimensions of the array.

pub fn metadata(&self) -> &CompressedFileMetadata

Get the metadata for the compressed file.

pub fn block_size(&self) -> usize

Get the block size.

pub fn num_blocks(&self) -> usize

Get the number of blocks.

pub fn preload_block(&self, blockidx: usize) -> CoreResult<()>

Load a specific block into memory.

This is useful for preloading blocks that will be accessed soon.

Arguments

• blockidx - The index of the block to load

Returns

Ok(()) if successful, or an error

pub fn readonly_array(&self) -> CoreResult<Array<A, IxDyn>>

Get a read-only view of the entire array.

This will decompress all blocks and load them into memory.

Returns

A read-only ndarray view of the data

pub fn get(&self, indices: &[usize]) -> CoreResult<A>

Get a specific element from the array.

This will decompress only the block containing the element.

Arguments

• indices - The indices of the element to get

Returns

The element at the specified indices

pub fn slice(&self, ranges: &[(usize, usize)]) -> CoreResult<Array<A, IxDyn>>

Get a subset of the array as a new array.

This will decompress only the blocks containing the subset.

Arguments

• ranges - The ranges of indices to get for each dimension

Returns

A new array containing the subset

pub fn process_blocks<F, R>(&self, f: F) -> CoreResult<Vec<R>>

where F: Fn(&[A], usize) -> R + Send + Sync + 'static, R: Send + 'static,

Process the array in blocks, with each block loaded and decompressed on demand.

This is useful for operations that can be performed on blocks independently.

Arguments

• f - A function that processes a block of elements

Returns

A vector of results, one for each block

pub fn process_blocks_with_size<F, R>( &self, blocksize: usize, f: F, ) -> CoreResult<Vec<R>>

where F: Fn(&[A], usize) -> R + Send + Sync + 'static, R: Send + 'static,

Process the array in blocks with a custom block size.

Arguments

• block_size - The block size to use (in elements)
• f - A function that processes a block of elements

Returns

A vector of results, one for each block

pub fn process_blocks_parallel<F, R>(&self, f: F) -> CoreResult<Vec<R>>

where F: Fn(&[A], usize) -> R + Send + Sync + 'static, R: Send + 'static,

Process the array in blocks in parallel.

Arguments

• f - A function that processes a block of elements

Returns

A vector of results, one for each block

pub fn process_blocks_parallel_with_size<F, R>( &self, block_size: usize, f: F, ) -> CoreResult<Vec<R>>

where F: Fn(&[A], usize) -> R + Send + Sync + 'static, R: Send + 'static,

Process the array in blocks in parallel with a custom block size.

Arguments

• block_size - The block size to use (in elements)
• f - A function that processes a block of elements

Returns

A vector of results, one for each block

impl<A: Clone + Copy + 'static + Send + Sync> CompressedMemMappedArray<A>

pub fn with_prefetching(self) -> PrefetchingCompressedArray<A>

Convert into a prefetching compressed array.

pub fn with_prefetching_config( self, config: PrefetchConfig, ) -> CoreResult<PrefetchingCompressedArray<A>>

Convert into a prefetching compressed array with the given configuration.

Trait Implementations

impl<A: Clone + Clone + Copy + 'static + Send + Sync> Clone for CompressedMemMappedArray<A>

fn clone(&self) -> CompressedMemMappedArray<A>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<A: Debug + Clone + Copy + 'static + Send + Sync> Debug for CompressedMemMappedArray<A>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.