Module memory_efficient 

Memory-efficient operations and views for large arrays.

This module provides utilities for working with large arrays efficiently by:

• Using chunk-wise processing to reduce memory requirements
• Creating memory-efficient views that avoid full data copies
• Implementing lazy evaluation and operation fusion for improved performance
• Supporting out-of-core processing for data that doesn’t fit in RAM
• Smart prefetching for improved performance with predictable access patterns

Modules

prelude

Structs

AdaptiveChunkingBuilder

Builder for creating adaptive chunking parameters with a fluent API.

AdaptiveChunkingParams

Parameters for configuring adaptive chunking behavior.

AdaptiveChunkingResult

Result of adaptive chunking analysis.

AdaptivePatternTracker

Advanced access pattern detector with dynamic learning.

AdaptivePrefetchConfig

Extended prefetching configuration with adaptive learning options.

AdaptivePrefetchConfigBuilder

Builder for adaptive prefetch configuration.

ArrayCreation

NumPy-compatible array creation functions

ArrayLayout

Array layout utilities for working with ndarray

BufferPool

Buffer pool for efficient memory management

BufferPoolStats

Statistics for buffer pool

ChunkIter

Iterator over chunks of a memory-mapped array (for 1D arrays only).

ChunkedArray

A chunked array that provides memory-efficient processing for large datasets

CompressedFileMetadata

Metadata for a compressed memory-mapped file

CompressedMemMapBuilder

Builder for compressed memory-mapped arrays.

CompressedMemMappedArray

A memory-mapped array with transparent compression.

CrossDeviceManager

Cross-device manager for handling data transfers and operations

CrossFilePrefetchConfig

Configuration for cross-file prefetching.

CrossFilePrefetchConfigBuilder

Builder for cross-file prefetch configuration.

CrossFilePrefetchManager

Manager for cross-file prefetching.

CrossFilePrefetchRegistry

A global registry for cross-file prefetching.

DataAccess

An access to a specific part of a dataset.

DataId

Unique identifier for zero-copy data

DataMetadata

Metadata about zero-copy data

DatasetId

A unique identifier for a file or array.

DeviceArray

Array residing on a specific device (CPU, GPU, TPU)

DeviceBuffer

Buffer for cross-device operations

DeviceMemoryManager

Device memory manager for cross-device operations

DeviceMemoryPool

Memory pool for efficient allocation on a device

DeviceStream

Stream for asynchronous operations

DiskBackedArray

A specialized out-of-core array that uses memory mapping for efficient access

InterfaceStats

Statistics for the zero-copy interface

LayoutConverter

Memory-efficient layout converter for large arrays

LazyArray

A lazy array that stores operations to be performed later

LazyOp

Represents an operation in the lazy evaluation graph

LockFreeQueue

Lock-free queue for zero-copy buffers

MemoryLayout

Memory layout information for arrays

MemoryMappedArray

Memory-mapped array that efficiently maps file data directly into memory

MemoryMappedSlice

A slice of a memory-mapped array that maintains memory-mapping.

NumaNode

NUMA node information for work-stealing optimization

NumaTopology

NUMA topology information

OpFusion

A structure for chaining multiple operations together and optimizing the execution

OutOfCoreArray

An array that stores data on disk to reduce memory usage

PatternRecognitionConfig

Configuration for pattern recognition.

PatternRecognizer

Pattern recognition engine for complex access patterns.

PatternTrackerFactory

Factory for creating different types of access pattern trackers.

Pipeline

Stream processing pipeline

PipelineBuilder

Pipeline builder

PipelineStats

Pipeline statistics

PrefetchConfig

Configuration for the prefetching system.

PrefetchConfigBuilder

Builder for prefetch configuration.

PrefetchStats

Statistics about prefetching performance.

PrefetchingCompressedArray

RecognizedPattern

A recognized pattern with metadata.

ResourceAwareConfig

Configuration for resource-aware prefetching.

ResourceAwareConfigBuilder

Builder for resource-aware configuration.

ResourceAwarePrefetcher

Resource-aware prefetching manager.

ResourceMonitor

Resource monitor for tracking system load.

ResourceSnapshot

Snapshot of system resource usage at a point in time.

ResourceSummary

Summary of resource usage over time.

SchedulerStats

Overall scheduler statistics

StreamConfig

Stream processor configuration

StreamConfigBuilder

Builder for stream processor configuration

StreamProcessor

Stream processor for continuous data flows

StreamStats

Stream processor statistics

TransferEvent

Event for tracking asynchronous operations

TransferOptions

Options for cross-device memory transfers

TransferOptionsBuilder

Builder for transfer options

WorkStealingConfig

Configuration for the work-stealing scheduler

WorkStealingConfigBuilder

Builder for work-stealing scheduler configuration

WorkStealingScheduler

Advanced work-stealing scheduler

ZeroCopyBuffer

Zero-copy buffer that manages memory without copying

ZeroCopyConfig

Configuration for zero-copy streaming

ZeroCopyData

Zero-copy data container that can be shared between components

ZeroCopyInterface

Zero-copy interface for data exchange between components

ZeroCopyStats

Statistics for zero-copy processing

ZeroCopyStreamProcessor

Zero-copy stream processor

ZeroCopyView

A view into a portion of zero-copy data

ZeroCopyWeakRef

Weak reference to zero-copy data

ZeroCopyWorkStealingScheduler

Work-stealing scheduler for efficient parallel processing

Enums

AccessMode

Access mode for memory-mapped arrays

AccessPattern

Types of access patterns that can be detected and prefetched.

AccessType

Types of data access.

ChunkingStrategy

Strategy for chunking arrays for memory-efficient processing

ComplexPattern

The different types of complex patterns that can be recognized.

CompressionAlgorithm

Supported compression algorithms

Confidence

Confidence level for pattern detection.

DeviceType

Device types supported by the cross-device memory management

KernelParam

Kernel parameter for GPU execution

LayoutOrder

Memory layout order following NumPy conventions

LazyOpKind

An enumeration of lazy operations that can be performed on arrays

MemoryAccessPattern

Access pattern for layout optimization

PrefetchStrategy

Types of prefetching strategies that can be dynamically selected.

ProcessingMode

Processing mode for zero-copy streaming

ResourceType

Types of system resources that can be monitored.

StreamMode

Stream processing mode

StreamSource

Input source for a data stream

StreamState

Stream processor state

TaskPriority

Task priority levels for the work-stealing scheduler

TransferDirection

Memory transfer direction between devices

TransferMode

Transfer mode for cross-device operations

WorkloadType

Beta 2: Workload types for optimized chunking strategies

Constants

OPTIMAL_CHUNK_SIZE

Optimal chunk size in bytes for memory-efficient operations Chosen as 16 MB which is a good trade-off between memory usage and performance

Traits

AdaptiveChunking

Trait for adaptive chunking capabilities.

ArithmeticOps

Extension trait for standard arithmetic operations on memory-mapped arrays.

BroadcastOps

Trait for broadcasting operations between memory-mapped arrays of different shapes.

DataExchange

Trait for types that can participate in zero-copy data exchange

DatasetPrefetcher

Interface for dataset-specific prefetchers.

FromZeroCopy

Helper trait for extracting data from zero-copy containers

FusedOp

A trait for operations that can be fused together for better performance

IntoZeroCopy

Helper trait for converting types to zero-copy data

MemoryMappedChunkIter

Extension trait for MemoryMappedArray to enable chunked iteration.

MemoryMappedChunks

Extension trait for MemoryMappedArray to enable chunked processing of large datasets.

MemoryMappedChunksParallel

Extension trait for parallel processing of memory-mapped arrays.

MemoryMappedSlicing

Extension trait for adding slicing functionality to MemoryMappedArray.

Prefetching

Trait for memory-mapped arrays that support prefetching.

ToDevice

Extension trait for arrays to simplify device transfers

ToHost

Extension trait for device arrays to simplify host transfers

WorkStealingTask

A task that can be executed by the work-stealing scheduler

ZeroCopyOps

Trait for zero-copy operations on memory-mapped arrays.

ZeroCopySerializable

Trait for zero-copy serializable types.

ZeroCopySerialization

An extension trait for MemoryMappedArray to support zero-copy serialization and deserialization.

ZeroCopyWorkStealingTask

Work-stealing task for the scheduler

Functions

chunk_wise_binary_op

Perform a binary operation on two arrays in a chunk-wise manner

chunk_wise_op

Perform an operation on an array in a chunk-wise manner to reduce memory usage

chunk_wise_reduce

Perform a reduction operation on an array in a chunk-wise manner

create_cpu_intensive_scheduler

Create a work-stealing scheduler optimized for CPU-intensive tasks

create_cpuarray

Create a device array on the CPU

create_cross_device_manager

Create a cross-device manager with default settings

create_disk_array

Create a disk-backed array from the given data and file path

create_global_data_registry

Create a global zero-copy data registry

create_gpuarray

Create a device array on the GPU

create_io_intensive_scheduler

Create a work-stealing scheduler optimized for I/O-intensive tasks

create_mmap

Create a new memory-mapped array file

create_pipeline

Create a new pipeline

create_stream_processor

Create a new stream processor with default configuration

create_temp_mmap

Create a new temporary memory-mapped array

create_work_stealing_scheduler

Create a default work-stealing scheduler

create_zero_copy_data

Create zero-copy data from a vector

create_zero_copy_processor

Create a new zero-copy stream processor with default configuration

diagonal_view

Create a copy of the diagonal of a 2D array

evaluate

Evaluate a lazy array and return a concrete array

get_global_data

Get data globally by name

global_interface

Get the global zero-copy interface

load_chunks

Load chunks of an out-of-core array into memory

open_mmap

Create a memory-mapped array from an existing file

register_fusion

Register a fused operation in the global registry

register_global_data

Register data globally by name

to_best_device

Transfer an array to the best available device

transpose_view

Create a transposed copy of a 2D array

view_as^⚠

Create a view of an array with a different element type

view_mut_as^⚠

Create a mutable view of an array with a different element type

Type Aliases

ArrayView

A type alias for ndarray’s ArrayView with additional functionality

ViewMut

A type alias for ndarray’s ArrayViewMut with additional functionality