Parcode

High-performance, zero-copy, lazy-loading object storage for Rust.

parcode is an architecture-aware storage system designed for complex, deep data structures. Unlike traditional serialization (JSON, Bincode) which treats data as a flat blob, parcode preserves the structure of your objects on disk.

This enables capabilities previously reserved for complex databases:

• Lazy Mirrors: Navigate deep struct hierarchies without loading data from disk.
• Surgical Access: Load only the specific field, vector chunk, or map entry you need.
• $O(1)$ Map Lookups: Retrieve items from huge HashMaps instantly without full deserialization.
• Parallel Speed: Writes are fully parallelized using a Zero-Copy graph architecture.

The Innovation: Pure Rust Lazy Loading

Most libraries that offer "Lazy Loading" or "Zero-Copy" access (like FlatBuffers or Cap'n Proto) come with a heavy price: Interface Definition Languages (IDLs). You are forced to write separate schema files (.proto, .fbs), run external compilers, and deal with generated code that doesn't feel like Rust.

Parcode changes the game.

We invented a technique we call "Native Mirroring". By simply adding #[derive(ParcodeObject)], Parcode analyzes your Rust structs at compile time and invisibly generates a Lazy Mirror API.

Installation

Add this to your Cargo.toml:

[dependencies]

parcode = "0.4.0"

To enable LZ4 compression:

[dependencies]

parcode = { version = "0.4.0", features = ["lz4_flex"] }

Usage Guide

1. Define your Data

Use #[derive(ParcodeObject)] and the #[parcode(...)] attributes to tell the engine how to shard your data.

use parcode::ParcodeObject; // Use ParcodeObject trait to enable lazy procedural macros
use serde::{Serialize, Deserialize};
use std::collections::HashMap;

#[derive(Serialize, Deserialize, ParcodeObject)]
struct GameWorld {
    id: u64,               // Stored Inline (Metadata)
    name: String,          // Stored Inline (Metadata)

    #[parcode(chunkable)]  // Stored in a separate, compressed chunk
    settings: WorldSettings,

    #[parcode(chunkable)]  // Automatically sharded into parallel chunks
    terrain: Vec<u8>, 
    
    #[parcode(map)]        // Hash-sharded for O(1) lookups
    players: HashMap<String, Player>,
}

#[derive(Serialize, Deserialize, ParcodeObject, Clone)]
struct WorldSettings {
    difficulty: u8,
    #[parcode(chunkable)]
    history: Vec<String>,
}

#[derive(Serialize, Deserialize, ParcodeObject, Clone)]
struct Player {
    level: u32,
    #[parcode(chunkable)]
    inventory: Vec<u32>, // Heavy data
}

2. Save Data

You have two ways to save data: Simple and Configured.

A. Simple Save (Default Settings) Perfect for quick prototyping.

use parcode::Parcode;

let world = GameWorld { /* ... */ };

// Saves with parallelism enabled, no compression
Parcode::save("savegame.par", &world)?;

B. Configured Save (Production) Use the builder to enable compression or write to arbitrary streams.

// Saves with LZ4 compression enabled
Parcode::builder()
    .compression(true)
    .write("savegame_compressed.par", &world)?;

3. Read Data (Lazy)

Here is where the magic happens. We don't load the object; we load a Mirror.

use parcode::ParcodeReader;

// 1. Open the file (Instant, uses mmap)
let reader = ParcodeReader::open("savegame.par")?;

// 2. Get the Lazy Mirror (Instant, reads only header)
// Note: We get 'GameWorldLazy', a generated shadow struct.
let world_mirror = reader.read_lazy::<GameWorld>()?;

// 3. Access local fields directly (Already in memory)
println!("World ID: {}", world_mirror.id);

// 4. Navigate hierarchy without I/O
// 'settings' is a mirror. Accessing it costs nothing.
// 'difficulty' is inline. Accessing it costs nothing.
println!("Difficulty: {}", world_mirror.settings.difficulty);

// 5. Surgical Load
// Only NOW do we touch disk to load the history vector.
// The massive 'terrain' vector is NEVER loaded.
let history = world_mirror.settings.history.load()?;

4. Advanced Access Patterns

O(1) Map Lookup

Retrieve a single user from a million-user database without loading the database.

// .get() returns a full object
// .get_lazy() returns a Mirror of the object!

if let Some(player_mirror) = world_mirror.players.get_lazy(&"Hero123".to_string())? {
    // Access player metadata instantly
    println!("Player Level: {}", player_mirror.level);
    
    // Only load inventory if needed
    let inv = player_mirror.inventory.load()?;
}

Lazy Vector Iteration

Scan a list of heavy objects without loading their heavy payloads.

// Assume we have Vec<Player>
for player_proxy in world_mirror.all_players.iter()? {
    let p = player_proxy?; // Resolve result
    
    // We can check level WITHOUT loading the player's inventory from disk!
    if p.level > 50 {
        println!("High level player found!");
        // p.inventory.load()?; 
    }
}

Advanced Features

Generic I/O: Write to Memory/Network

Parcode isn't limited to files. You can serialize directly to any std::io::Write destination.

let mut buffer = Vec::new();

// Serialize directly to RAM
Parcode::builder()
    .compression(true)
    .write_to_writer(&mut buffer, &my_data)?;

// 'buffer' now contains the full Parcode file structure

Synchronous Mode

For environments where threading is not available (WASM, embedded) or to reduce memory overhead.

Parcode::builder()
    .compression(true)
    .write_sync("sync_save.par", &data)?;

Forensic Inspector

Parcode includes tools to analyze the structure of your files without deserializing them.

use parcode::inspector::ParcodeInspector;

let report = ParcodeInspector::inspect("savegame.par")?;
println!("{}", report);

Output:

=== PARCODE INSPECTOR REPORT ===
Root Offset:    550368
[GRAPH LAYOUT]
└── [Generic Container] Size: 1b | Algo: None | Children: 2
    ├── [Vec Container] Size: 13b | Algo: LZ4 | Children: 32 [Vec<50000> items] 
    └── [Map Container] Size: 4b | Algo: None | Children: 4 [Hashtable with 4 buckets]

Macro Attributes Reference

Control exactly how your data structure maps to disk using #[parcode(...)].

Benchmarks vs The World

Scenario: Cold Start of an application reading a massive World State file (100MB+).

Benchmarks run on NVMe SSD. Parallel throughput scales with cores.

License

This project is licensed under the MIT license.

Built for the Rust community by RetypeOS.