# memmap3
[](https://crates.io/crates/memmap3)
[](https://docs.rs/memmap3)
Safe memory-mapped I/O with zero-copy persistent data structures.
**memmap3** is a complete drop-in replacement for [memmap2](https://docs.rs/memmap2) that adds a
huge additional list of 'safe' features. All memmap2 functionality is available, plus powerful new
APIs that eliminate unsafe code while providing automatic persistence.
## 🎯 Core Features Summary
- **🔄 Auto-Persistence**: Changes persist automatically via memory mapping
- **🔒 Zero Unsafe**: Safe Rust APIs for all operations
- **⚡ Thread-Safe**: Atomic fields work across processes
- **🎭 Convenience Operators**: `<<` for intuitive data manipulation
- **📏 Fixed-Size + Unlimited**: Fixed layouts + segmented growth when needed
- **🧩 Rich Type System**: Primitives, atomics, strings, collections, nested structs
- **📊 Multidimensional**: N-dimensional arrays for join-like operations
- **🔌 Drop-in Compatible**: All memmap2 code works unchanged
## Quick Start
Add to your `Cargo.toml`:
```toml
[dependencies]
memmap3 = "0.1"
```
Create persistent data structures:
```rust
use memmap3::prelude::*;
#[mmap_struct]
struct Counter {
#[mmap(atomic)]
value: u64,
name: [u8; 32],
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Create persistent data
let mut counter = MmapStruct::<Counter>::create("/tmp/counter.mmap")?;
&mut counter.name << "my_counter";
// Atomic operations work across processes
let old_value = counter.value.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
println!("Counter: {} -> {}", old_value, old_value + 1);
Ok(())
}
```
## Key Features
### 🔒 Safe API
- **Zero unsafe code** required in user applications
- **Compile-time validation** through `#[mmap_struct]` macro
- **Runtime bounds checking** for all operations
- **UTF-8 validation** for string operations
### ⚡ High Performance
- **Zero-copy access** to persistent data
- **OS-level caching** through memory mapping
- **Atomic operations** for thread-safe access
- **Minimal overhead** with direct memory access
### 🧩 Drop-in Compatibility
```rust
// Just change your import:
use memmap2::{Mmap, MmapOptions}; // From this
use memmap3::{Mmap, MmapOptions}; // To this
// All existing memmap2 code works unchanged!
```
## 🔧 The #[mmap_struct] Macro
Transform regular Rust structs into persistent, memory-mapped types:
```rust
use memmap3::prelude::*;
#[mmap_struct]
struct MyData {
// Regular fields (primitives, enums)
id: u64,
active: bool,
// Atomic fields (thread-safe across processes)
#[mmap(atomic)]
counter: u64,
// Auto-detected strings (default for [u8; N])
name: [u8; 32],
// Explicit raw binary data
#[mmap(raw)]
key: [u8; 32],
// Fixed-capacity vectors
#[mmap(vec)]
scores: [u32; 10],
// String arrays
#[mmap(string_array)]
tags: [[u8; 16]; 5],
// Unlimited growth storage
#[mmap(segmented)]
events: [u64; 0],
}
```
## 🎭 Convenience Operators
Smooth, intuitive syntax for common operations:
```rust
// String assignment
&mut my_string << "Hello";
// Vector append
&mut my_vec << item;
// HashMap insertion
&mut my_map << ("key", "value");
// HashMap key assignment using macro
hashmap_index!(my_map["key"] << "value");
// Segmented append
&mut my_segmented << item;
```
## 📊 Complete Type System
### Primitive Types (Native Support)
- **Integers**: u8, u16, u32, u64, u128, i8, i16, i32, i64, i128, usize, isize
- **Floats**: f32, f64
- **Bool**: bool
- **Char**: char (stored as u32)
- **C-style Enums**: Any enum with `#[repr(C)]`
### Atomic Types (`#[mmap(atomic)]`)
Transform any primitive to thread-safe atomic:
- `u64` → `MmapAtomicU64` (fetch_add, compare_exchange, etc.)
- `bool` → `MmapAtomicBool` (load, store, etc.)
- Plus all integer types + usize/isize
### String Types
- **Auto-detected**: `[u8; N]` → `MmapString<N>` (default behavior)
- **Explicit**: `#[mmap(string)]` to force string behavior
- **Raw binary**: `#[mmap(raw)]` to keep as byte array
### Collection Types
- **Vectors**: `#[mmap(vec)]` transforms `[T; N]` → `MmapVec<T, N>`
- **String Arrays**: `#[mmap(string_array)]` transforms `[[u8; LEN]; N]` → `MmapStringArray<N, LEN>`
- **Hash Maps**: `MmapHashMap<K, V>` for key-value storage
### Advanced Types
- **Segmented**: `#[mmap(segmented)]` for unlimited growth (breaks fixed-size constraint)
- **Nested Structs**: Full support for complex nested structures
- **Multidimensional Arrays**: `[[[u8; X]; Y]; Z]` for N-dimensional data
- **Serde Integration**: Store any `Serialize/Deserialize` type
## 🔗 Multidimensional & Join-like Operations
Create sophisticated data structures for complex queries:
```rust
use memmap3::prelude::*;
#[mmap_struct]
struct BitIndex {
// 3D lookup table for fast joins
index: [[[u8; 256]; 256]; 256],
// User data indexed by multiple dimensions
users_by_age_location: [[u64; 100]; 50], // [age][location] → user_id
// Unlimited growth using segmented
#[mmap(segmented)]
overflow_data: [u64; 0],
}
```
## 🏗️ Fixed-Size Constraint + Segmented Growth
- **Most types**: Fixed size at compile time for predictable layout
- **Exception**: `#[mmap(segmented)]` allows unlimited growth
- **Benefit**: Known memory layout enables direct pointer arithmetic
## ⚙️ How Auto-Persistence Works
1. **Memory Mapping**: File mapped into process memory
2. **Direct Access**: Struct fields map directly to file bytes
3. **OS Sync**: Operating system handles sync to disk
4. **Cross-Process**: Multiple processes can share same file safely
## Core Types
### MmapStruct<T>
Safe wrapper for memory-mapped structs with automatic file management:
```rust
use memmap3::prelude::*;
#[mmap_struct]
struct GameState {
level: u32,
score: u64,
player_name: [u8; 32],
}
let mut game = MmapStruct::<GameState>::create("game.mmap")?;
game.level = 5;
game.score = 12345;
&mut game.player_name << "Player1";
```
### Atomic Fields
Thread-safe operations across processes:
```rust
use memmap3::prelude::*;
#[mmap_struct]
struct Metrics {
#[mmap(atomic)]
requests: u64,
#[mmap(atomic)]
errors: u64,
}
let metrics = MmapStruct::<Metrics>::create("metrics.mmap")?;
metrics.requests.fetch_add(1, Ordering::SeqCst);
```
### Collections
Persistent data structures with dynamic sizing:
```rust
use memmap3::prelude::*;
// Persistent hash map
let mut cache = MmapHashMap::<&str, &str>::create("cache.mmap")?;
cache.insert("key1", "value1")?;
// Multiple assignment syntaxes available:
&mut cache << ("key2", "value2"); // Tuple insertion
hashmap_index!(cache["key3"] << "value3"); // Index-style assignment
cache.set("key4", "value4")?; // Direct set method
// Persistent vector
let mut log = MmapVec::<u64, 1000>::new();
log.push(1234567890); // timestamp
// Persistent string
let mut name = MmapString::<64>::new();
&mut name << "Alice";
```
## Supported Types
### Primitive Types (no attribute needed)
- **Integers**: u8, u16, u32, u64, u128, i8, i16, i32, i64, i128, usize, isize
- **Floats**: f32, f64
- **Bool**: bool
- **Char**: char (stored as u32)
- **C-style Enums**: Any enum with `#[repr(C)]`
### Transformed Types (via attributes)
#### Atomic Types (`#[mmap(atomic)]`)
Transforms regular types to atomic equivalents for thread-safe access:
- `u8` → `MmapAtomicU8`, `u16` → `MmapAtomicU16`, etc.
- `bool` → `MmapAtomicBool`
#### Strings (`#[mmap(string)]`)
Transforms byte arrays to UTF-8 strings with helper methods:
- `[u8; N]` → `MmapString<N>`
- Provides `.set()`, `<<` operator, `.as_str()`, `.clear()`
#### Vectors (`#[mmap(vec)]`)
Transforms arrays to fixed-capacity vectors:
- `[T; N]` → `MmapVec<T, N>`
- Provides `.push()`, `.pop()`, `.len()`, `.iter()`, indexing
#### String Arrays (`#[mmap(string_array)]`)
Transforms 2D byte arrays to arrays of strings:
- `[[u8; LEN]; N]` → `MmapStringArray<N, LEN>`
- Provides indexed access to multiple strings
## Running Examples
See the `examples/` directory for complete working examples:
```bash
cargo run --example 01-002_macro_usage
cargo run --example 03-001_atomic_operations
cargo run --example 02-002_string_operations
cargo run --example 04-003_event_log
cargo run --example 14_hashmap
```
## Examples
### Simple Configuration Storage
```rust
use memmap3::prelude::*;
#[mmap_struct]
struct Config {
max_connections: u32,
timeout_ms: u32,
debug_mode: bool,
server_name: [u8; 64],
}
let mut config = MmapStruct::<Config>::create("app.config")?;
config.max_connections = 1000;
config.timeout_ms = 5000;
&mut config.server_name << "production-server";
// Configuration persists across application restarts
```
### Inter-Process Communication
```rust
use memmap3::prelude::*;
use std::sync::atomic::Ordering;
#[mmap_struct]
struct SharedData {
#[mmap(atomic)]
message_count: u64,
#[mmap(atomic)]
worker_status: u32,
shared_buffer: [u8; 1024],
}
// Process A
let shared = MmapStruct::<SharedData>::create("/tmp/ipc.mmap")?;
shared.message_count.store(42, Ordering::SeqCst);
// Process B can read the same data
let shared = MmapStruct::<SharedData>::open("/tmp/ipc.mmap")?;
let count = shared.message_count.load(Ordering::SeqCst);
println!("Messages: {}", count); // Prints: Messages: 42
```
### Persistent Analytics
```rust
use memmap3::prelude::*;
use std::sync::atomic::Ordering;
#[mmap_struct]
struct Analytics {
#[mmap(atomic)]
page_views: u64,
#[mmap(atomic)]
unique_visitors: u64,
#[mmap(atomic)]
conversion_rate: u64, // Fixed-point arithmetic
}
let analytics = MmapStruct::<Analytics>::create("analytics.mmap")?;
// Record events
analytics.page_views.fetch_add(1, Ordering::SeqCst);
analytics.unique_visitors.fetch_add(1, Ordering::SeqCst);
// Calculate conversion rate (example: 2.5% as 250 basis points)
let rate = (conversions * 10000) / page_views;
analytics.conversion_rate.store(rate, Ordering::SeqCst);
```
## How It Works
The `#[mmap_struct]` attribute macro:
1. Automatically adds `#[repr(C)]` for predictable memory layout
2. Transforms annotated fields to thread-safe types
3. Generates trait implementations for memory mapping
4. Handles file creation, opening, and validation
## Thread Safety
- **Regular fields**: Single writer, multiple readers (protected by OS page cache)
- **Atomic fields**: Multiple concurrent writers and readers
- **String fields**: Single writer with UTF-8 validation
### Using with Arc for Shared Access
You can wrap `MmapStruct` in `Arc` for safe sharing across threads:
```rust
use memmap3::prelude::*;
use std::sync::Arc;
use std::sync::atomic::Ordering;
#[mmap_struct]
struct Config {
port: u16,
#[mmap(atomic)]
counter: u64,
}
// Single-threaded use
let mut data = MmapStruct::<Config>::create("config.mmap")?;
data.port = 8080; // Can modify all fields
// Multi-threaded use with Arc
let shared = Arc::new(MmapStruct::<Config>::create("config.mmap")?);
// Clone for each thread
let thread_data = shared.clone();
thread_data.counter.fetch_add(1, Ordering::Relaxed);
// Regular fields are read-only when shared via Arc
println!("Port: {}", thread_data.port);
});
```
**Access Patterns:**
- **No Arc**: Full read/write access to all fields (single-threaded)
- **With Arc**: Atomic fields writable, regular fields read-only (multi-threaded)
- **With Arc<Mutex<...>>**: If you need to modify regular fields from multiple threads
## String Handling
Automatic string operations for byte arrays:
```rust
use memmap3::prelude::*;
#[mmap_struct]
struct User {
id: u64,
name: [u8; 32], // Automatically treated as string
email: [u8; 128], // Automatically treated as string
}
let mut user = MmapStruct::<User>::create("user.mmap")?;
user.id = 12345;
// Write strings using << operator
&mut user.name << "Alice Smith";
&mut user.email << "alice@example.com";
// Read strings back
println!("User: {} ({})", user.name.as_str(), user.email.as_str());
```
## Feature Flags
Enable additional functionality:
```toml
[dependencies]
memmap3 = { version = "0.1", features = ["serde"] }
```
- `serde` - Serialize/deserialize support for complex types
## Documentation
- **[API Reference](https://docs.rs/memmap3)** - Complete API documentation
- **[Cookbook Examples](cookbooks/)** - Real-world usage patterns
- **[Migration Guide](https://github.com/deepbrainspace/memmap3#migration-from-memmap2)** -
Upgrading from memmap2
## Use Cases
- **Inter-process communication** - Shared configuration and state
- **Event logging** - High-performance ring buffers
- **ML model weights** - Instant loading of large models
- **Game save states** - Fast persistence and loading
- **Database indexes** - Memory-mapped B-trees
## Performance
memmap3 provides zero-copy access with minimal overhead:
- **Direct memory access** - No serialization/deserialization
- **OS page cache integration** - Automatic memory management
- **Lazy loading** - Only accessed pages are loaded
- **Cross-process sharing** - Efficient memory usage
## Safety Guarantees
Multiple layers of safety validation:
1. **Compile-time validation** - `#[mmap_struct]` macro checks
2. **Runtime bounds checking** - All array/collection access
3. **Memory layout verification** - Struct compatibility validation
4. **UTF-8 validation** - String operation safety
5. **Atomic operation safety** - Cross-process synchronization
## License
This project is licensed under the MIT OR Apache-2.0 license.
## Contributing
Contributions are welcome! Please see our
[GitHub repository](https://github.com/deepbrainspace/memmap3) for guidelines.