Module reader 

The Read-Side Engine: Parallel Reconstruction & Random Access.

This module implements the complete reading pipeline for Parcode files, providing both eager (full deserialization) and lazy (on-demand) loading strategies. It leverages memory mapping, parallel reconstruction, and zero-copy techniques to maximize performance.

Core Architecture

The reader is built on three foundational techniques:

1. Memory Mapping (mmap)

Instead of reading the entire file into memory, Parcode uses mmap to map the file directly into the process’s address space. This provides several benefits:

• Instant Startup: Opening a file is O(1) regardless of size
• OS-Managed Paging: The operating system handles loading pages on demand
• Zero-Copy Reads: Uncompressed data can be read directly from the mapped region
• Shared Memory: Multiple processes can share the same mapped file

2. Lazy Traversal

The file is traversed lazily - we only read and decompress bytes when a specific node is requested. This enables:

• Cold Start Performance: Applications can start in microseconds
• Selective Loading: Load only the data you need
• Deep Navigation: Traverse object hierarchies without I/O

3. Parallel Zero-Copy Stitching

When reconstructing large Vec<T>, we use a sophisticated parallel algorithm:

┌─────────────────────────────────────────────────────────────┐
│ 1. Pre-allocate uninitialized buffer (MaybeUninit<T>)       │
├─────────────────────────────────────────────────────────────┤
│ 2. Calculate destination offset for each shard              │
├─────────────────────────────────────────────────────────────┤
│ 3. Spawn parallel workers (Rayon)                           │
├─────────────────────────────────────────────────────────────┤
│ 4. Each worker:                                             │
│    - Decompresses its shard                                 │
│    - Deserializes items                                     │
│    - Writes directly to final buffer (ptr::copy)            │
├─────────────────────────────────────────────────────────────┤
│ 5. Transmute buffer to Vec<T> (all items initialized)       │
└─────────────────────────────────────────────────────────────┘

Result: Maximum memory bandwidth, zero intermediate allocations, perfect parallelism.

O(1) Arithmetic Navigation

Using the RLE (Run-Length Encoding) metadata stored in container nodes, we can calculate exactly which physical chunk holds the Nth item of a collection. This enables:

• Random Access: vec.get(1_000_000) without loading the entire vector
• Constant Time: O(1) shard selection via arithmetic
• Minimal I/O: Load only the shard containing the target item

Trait System for Strategy Selection

The module defines two key traits that enable automatic strategy selection:

ParcodeNative

Types implementing this trait know how to reconstruct themselves from a ChunkNode. The high-level API (Parcode::load) uses this trait to automatically select the optimal reconstruction strategy:

• Vec<T>: Uses parallel reconstruction across shards
• HashMap<K, V>: Reconstructs all shards and merges entries
• Primitives/Structs: Uses sequential deserialization

ParcodeItem

Types implementing this trait can be read from a shard (payload + children). This trait is used internally during parallel reconstruction to deserialize individual items or slices of items from shard payloads.

Usage Patterns

Eager Loading (Full Deserialization)

use parcode::Parcode;

// Load entire object into memory
let data = vec![1, 2, 3];
Parcode::save("numbers_reader.par", &data).unwrap();
let data: Vec<i32> = Parcode::load("numbers_reader.par").unwrap();

Lazy Loading (On-Demand)

use parcode::{Parcode, ParcodeObject};
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, ParcodeObject)]
struct Assets {
    #[parcode(chunkable)]
    data: Vec<u8> }

#[derive(Serialize, Deserialize, ParcodeObject)]
struct GameState {
    level: u32,
    #[parcode(chunkable)]
    assets: Assets,
}

// Setup
let state = GameState { level: 1, assets: Assets { data: vec![0; 10] } };
Parcode::save("game_reader.par", &state).unwrap();

let file = Parcode::open("game_reader.par").unwrap();
let game_lazy = file.root::<GameState>().unwrap();

// Access local fields (instant, already in memory)
println!("Level: {}", game_lazy.level);

// Load remote fields on demand
let assets_data = game_lazy.assets.data.load().unwrap();

Random Access

use parcode::{Parcode, ParcodeObject};
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, ParcodeObject, Clone, Debug)]
struct MyStruct { val: u32 }

// Setup
let data: Vec<MyStruct> = (0..100).map(|i| MyStruct { val: i }).collect();
Parcode::save("data_random.par", &data).unwrap();

let file = Parcode::open("data_random.par").unwrap();
let root = file.root::<Vec<MyStruct>>().unwrap();

// Get item at index 50 without loading the entire vector
// Note: Using 50 instead of 1,000,000 for a realistic small test
let item = root.get(50).unwrap();

Streaming Iteration

use parcode::{Parcode, ParcodeObject};
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, ParcodeObject, Clone, Debug)]
struct MyStruct { val: u32 }

fn process(item: MyStruct) { println!("{:?}", item); }

// Setup
let data: Vec<MyStruct> = (0..10).map(|i| MyStruct { val: i }).collect();
Parcode::save("data_iter.par", &data).unwrap();

let file = Parcode::open("data_iter.par").unwrap();
let items: Vec<MyStruct> = file.load().unwrap();

// Note: The current API doesn't have a direct `iter` on root for Vecs yet,
// it usually goes through read_lazy or decode.
// Assuming we just decode for now as the example implies iteration capability.
for item in items {
    process(item);
}

Performance Characteristics

• File Opening: O(1) - just maps the file
• Root Access: O(1) - reads only the global header
• Random Access: O(1) - arithmetic shard selection + single shard load
• Parallel Reconstruction: O(N/cores) - scales linearly with CPU cores
• Memory Usage (Lazy): O(accessed chunks) - only loaded data consumes RAM
• Memory Usage (Eager): O(N) - entire object in memory

Thread Safety

• ParcodeFile: Cheap to clone (Arc-based), safe to share across threads
• ChunkNode: Immutable view, safe to share across threads
• Parallel Reconstruction: Uses Rayon’s work-stealing scheduler

Safety Considerations

The module uses unsafe code in two specific contexts:

1. Memory Mapping: mmap is inherently unsafe if the file is modified externally. We assume files are immutable during reading.

2. Parallel Stitching: Uses MaybeUninit and pointer arithmetic to avoid initialization overhead. All unsafe operations are carefully encapsulated and documented with safety invariants.

Structs

ChunkIterator

An iterator that loads shards on demand, allowing iteration over datasets larger than available RAM.

ChunkNode

A lightweight cursor pointing to a specific node in the dependency graph.

ParcodeFile

Represents an open Parcode file mapped in memory.

Traits

ParcodeItem

A trait for types that can be read from a shard (payload + children).

ParcodeNative

A trait for types that know how to reconstruct themselves from a ChunkNode.