# Parcode
[](https://crates.io/crates/parcode)
[](https://docs.rs/parcode)
[](https://github.com/retypeos/parcode/actions/workflows/ci.yml)
[](./LICENSE)
---
**High-performance, zero-copy, lazy-loading object storage for Rust.**
`parcode` is an architecture-aware storage system designed for complex, deep data structures. Unlike traditional serialization (JSON, Bincode) which treats data as a flat blob, `parcode` preserves the **structure** of your objects on disk.
This enables capabilities previously reserved for complex databases:
* **Lazy Mirrors:** Navigate deep struct hierarchies without loading data from disk.
* **Surgical Access:** Load only the specific field, vector chunk, or map entry you need.
* **$O(1)$ Map Lookups:** Retrieve items from huge `HashMap`s instantly without full deserialization.
* **Parallel Speed:** Writes are fully parallelized using a Zero-Copy graph architecture.
---
## The Innovation: Pure Rust Lazy Loading
Most libraries that offer "Lazy Loading" or "Zero-Copy" access (like FlatBuffers or Cap'n Proto) come with a heavy price: **Interface Definition Languages (IDLs)**. You are forced to write separate schema files (`.proto`, `.fbs`), run external compilers, and deal with generated code that doesn't feel like Rust.
**Parcode changes the game.**
We invented a technique we call **"Native Mirroring"**. By simply adding `#[derive(ParcodeObject)]`, Parcode analyzes your Rust structs at compile time and invisibly generates a **Lazy Mirror** API.
| **Schema Definition** | External IDL files (`.fbs`) | **Standard Rust Structs** |
| **Build Process** | Requires external CLI (`flatc`) | **Standard `cargo build`** |
| **Refactoring** | Manual sync across files | **IDE Rename / Refactor** |
| **Developer Experience** | Foreign | **Native** |
## Installation
Add this to your `Cargo.toml`:
```toml
[dependencies]
parcode = "0.4.0"
```
To enable LZ4 compression:
```toml
[dependencies]
parcode = { version = "0.4.0", features = ["lz4_flex"] }
```
---
## Usage Guide
### 1. Define your Data
Use `#[derive(ParcodeObject)]` and the `#[parcode(...)]` attributes to tell the engine how to shard your data.
```rust
use parcode::ParcodeObject; // Use ParcodeObject trait to enable lazy procedural macros
use serde::{Serialize, Deserialize};
use std::collections::HashMap;
#[derive(Serialize, Deserialize, ParcodeObject)]
struct GameWorld {
id: u64, // Stored Inline (Metadata)
name: String, // Stored Inline (Metadata)
#[parcode(chunkable)] // Stored in a separate, compressed chunk
settings: WorldSettings,
#[parcode(chunkable)] // Automatically sharded into parallel chunks
terrain: Vec<u8>,
#[parcode(map)] // Hash-sharded for O(1) lookups
players: HashMap<String, Player>,
}
#[derive(Serialize, Deserialize, ParcodeObject, Clone)]
struct WorldSettings {
difficulty: u8,
#[parcode(chunkable)]
history: Vec<String>,
}
#[derive(Serialize, Deserialize, ParcodeObject, Clone)]
struct Player {
level: u32,
#[parcode(chunkable)]
inventory: Vec<u32>, // Heavy data
}
```
### 2. Save Data
You have two ways to save data: Simple and Configured.
**A. Simple Save (Default Settings)**
Perfect for quick prototyping.
```rust
use parcode::Parcode;
let world = GameWorld { /* ... */ };
// Saves with parallelism enabled, no compression
Parcode::save("savegame.par", &world)?;
```
**B. Configured Save (Production)**
Use the builder to enable compression or write to arbitrary streams.
```rust
// Saves with LZ4 compression enabled
Parcode::builder()
.compression(true)
.write("savegame_compressed.par", &world)?;
```
### 3. Read Data (Lazy)
Here is where the magic happens. We don't load the object; we load a **Mirror**.
```rust
use parcode::ParcodeReader;
// 1. Open the file (Instant, uses mmap)
let reader = ParcodeReader::open("savegame.par")?;
// 2. Get the Lazy Mirror (Instant, reads only header)
// Note: We get 'GameWorldLazy', a generated shadow struct.
let world_mirror = reader.read_lazy::<GameWorld>()?;
// 3. Access local fields directly (Already in memory)
println!("World ID: {}", world_mirror.id);
// 4. Navigate hierarchy without I/O
// 'settings' is a mirror. Accessing it costs nothing.
// 'difficulty' is inline. Accessing it costs nothing.
println!("Difficulty: {}", world_mirror.settings.difficulty);
// 5. Surgical Load
// Only NOW do we touch disk to load the history vector.
// The massive 'terrain' vector is NEVER loaded.
let history = world_mirror.settings.history.load()?;
```
### 4. Advanced Access Patterns
#### O(1) Map Lookup
Retrieve a single user from a million-user database without loading the database.
```rust
// .get() returns a full object
// .get_lazy() returns a Mirror of the object!
if let Some(player_mirror) = world_mirror.players.get_lazy(&"Hero123".to_string())? {
// Access player metadata instantly
println!("Player Level: {}", player_mirror.level);
// Only load inventory if needed
let inv = player_mirror.inventory.load()?;
}
```
#### Lazy Vector Iteration
Scan a list of heavy objects without loading their heavy payloads.
```rust
// Assume we have Vec<Player>
for player_proxy in world_mirror.all_players.iter()? {
let p = player_proxy?; // Resolve result
// We can check level WITHOUT loading the player's inventory from disk!
if p.level > 50 {
println!("High level player found!");
// p.inventory.load()?;
}
}
```
---
## Advanced Features
### Generic I/O: Write to Memory/Network
Parcode isn't limited to files. You can serialize directly to any `std::io::Write` destination.
```rust
let mut buffer = Vec::new();
// Serialize directly to RAM
Parcode::builder()
.compression(true)
.write_to_writer(&mut buffer, &my_data)?;
// 'buffer' now contains the full Parcode file structure
```
### Synchronous Mode
For environments where threading is not available (WASM, embedded) or to reduce memory overhead.
```rust
Parcode::builder()
.compression(true)
.write_sync("sync_save.par", &data)?;
```
### Forensic Inspector
Parcode includes tools to analyze the structure of your files without deserializing them.
```rust
use parcode::inspector::ParcodeInspector;
let report = ParcodeInspector::inspect("savegame.par")?;
println!("{}", report);
```
**Output:**
```text
=== PARCODE INSPECTOR REPORT ===
Root Offset: 550368
[GRAPH LAYOUT]
└── [Map Container] Size: 4b | Algo: None | Children: 4 [Hashtable with 4 buckets]
```
---
## Macro Attributes Reference
Control exactly how your data structure maps to disk using `#[parcode(...)]`.
| **(none)** | Field is serialized into the parent's payload. | Small primitives (`u32`, `bool`), short Strings, flags. |
| `#[parcode(chunkable)]` | Field is stored in its own independent Chunk. | Structs, Vectors, or fields you want to load lazily (`.load()`). |
| `#[parcode(map)]` | Field (`HashMap`) is sharded by hash. | Large Dictionaries/Indices where you need random access (`.get()`). |
| `#[parcode(compression="lz4")]` | Overrides compression for this chunk. | Highly compressible data (text, save states). |
---
## Benchmarks vs The World
> **Scenario:** Cold Start of an application reading a massive World State file (100MB+).
| **Cold Start** (Ready to read metadata) | **Parcode** | **0.16 ms** | **0 MB** | Instant. Only headers read. |
| | Bincode | 97.47 ms | 30 MB | Forced to deserialize everything. |
| **Deep Fetch** (Load 1 asset) | **Parcode** | **3.20 ms** | **3.8 MB** | Loads only the target 1MB chunk. |
| | Bincode | 97.47 ms | 30 MB | Same cost as full load. |
| **Map Lookup** (Find user by ID) | **Parcode** | **0.02 ms** | **0 MB** | **4000x Faster**. Hash Sharding win. |
*Benchmarks run on NVMe SSD. Parallel throughput scales with cores.*
---
## License
This project is licensed under the [MIT license](LICENSE).
*Built for the Rust community by RetypeOS.*