Trait MemoryMappedChunks 

pub trait MemoryMappedChunks<A: Clone + Copy + 'static + Send + Sync> {
    // Required methods
    fn chunk_count(&self, strategy: ChunkingStrategy) -> usize;
    fn process_chunks<F, R>(&self, strategy: ChunkingStrategy, f: F) -> Vec<R>
       where F: Fn(&[A], usize) -> R;
    fn process_chunks_mut<F>(&mut self, strategy: ChunkingStrategy, f: F)
       where F: Fn(&mut [A], usize);
}

Extension trait for MemoryMappedArray to enable chunked processing of large datasets.

This trait extends MemoryMappedArray with methods for processing large datasets in manageable chunks, which helps to control memory usage and enables working with arrays that might be too large to fit entirely in memory.

Required Methods

fn chunk_count(&self, strategy: ChunkingStrategy) -> usize

Get the number of chunks for the given chunking strategy.

Arguments

• strategy - The chunking strategy to determine chunk sizes

Returns

The number of chunks that the array will be divided into

Examples

use scirs2_core::memory_efficient::{create_mmap, AccessMode, ChunkingStrategy, MemoryMappedChunks};
use scirs2_core::ndarray::Array1;

// Create a memory-mapped array with 100 elements
let data = Array1::<f64>::linspace(0., 99., 100);
let file_path = "example.bin";  // In practice, use a proper temporary path
let mmap = create_mmap(&data, file_path.as_ref(), AccessMode::Write, 0).expect("Operation failed");

// Check how many chunks we'll get with different strategies
assert_eq!(mmap.chunk_count(ChunkingStrategy::Fixed(10)), 10);  // 10 chunks of 10 elements each
assert_eq!(mmap.chunk_count(ChunkingStrategy::NumChunks(5)), 5);  // 5 chunks of 20 elements each

fn process_chunks<F, R>(&self, strategy: ChunkingStrategy, f: F) -> Vec<R>

where F: Fn(&[A], usize) -> R,

Process each chunk with a function and collect the results.

This method divides the array into chunks according to the given strategy, applies the provided function to each chunk, and collects the results into a vector. It is efficient for read-only operations on large arrays.

Arguments

• strategy - The chunking strategy to determine chunk sizes
• f - A function that processes each chunk and returns a result

Returns

A vector containing the results from processing each chunk

Examples

use scirs2_core::memory_efficient::{create_mmap, AccessMode, ChunkingStrategy, MemoryMappedChunks};
use scirs2_core::ndarray::Array1;

// Create a memory-mapped array with 20 elements (small numbers to avoid overflow)
let data = Array1::<f64>::from_vec((0..20).map(|x| x as f64).collect());
let file_path = "example.bin";  // In practice, use a proper temporary path
let mmap = create_mmap(&data, file_path.as_ref(), AccessMode::Write, 0).expect("Operation failed");

// Calculate the sum of each chunk
let chunk_sums = mmap.process_chunks(
    ChunkingStrategy::Fixed(5),
    |chunk, chunk_idx| chunk.iter().sum::<f64>()
);

// We should have 4 chunks with sums of elements 0-4, 5-9, 10-14, 15-19
assert_eq!(chunk_sums.len(), 4);

fn process_chunks_mut<F>(&mut self, strategy: ChunkingStrategy, f: F)

where F: Fn(&mut [A], usize),

Process each chunk with a mutable function that modifies the data in-place.

This method divides the array into chunks according to the given strategy and applies the provided mutable function to each chunk. The function can modify the chunk data in-place, and the changes will be saved to the underlying memory-mapped file.

Arguments

• strategy - The chunking strategy to determine chunk sizes
• f - A function that processes and potentially modifies each chunk

Examples

use scirs2_core::memory_efficient::{create_mmap, AccessMode, ChunkingStrategy, MemoryMappedChunks};
use scirs2_core::ndarray::Array1;

// Create a memory-mapped array with 100 zeros
let data = Array1::<f64>::zeros(100);
let file_path = "example.bin";  // In practice, use a proper temporary path
let mut mmap = create_mmap(&data, file_path.as_ref(), AccessMode::Write, 0).expect("Operation failed");

// Modify each chunk: set elements to their index
mmap.process_chunks_mut(
    ChunkingStrategy::Fixed(10),
    |chunk, chunk_idx| {
        for (i, elem) in chunk.iter_mut().enumerate() {
            *elem = (chunk_idx * 10 + i) as f64;
        }
    }
);

// Now the array contains [0, 1, 2, ..., 99]

Notes

This method uses direct file I/O to ensure changes are properly persisted to disk, which may be slower than memory-only operations but is more reliable for ensuring data is properly saved, especially with large datasets.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors

impl<A: Clone + Copy + 'static + Send + Sync> MemoryMappedChunks<A> for MemoryMappedArray<A>