Crate memmap3

Expand description

Safe memory-mapped I/O with zero-copy persistent data structures

memmap3 is a complete drop-in replacement for memmap2 that adds a huge additional list of ‘safe’ features. All memmap2 functionality is available, plus powerful new APIs that eliminate unsafe code while providing automatic persistence.

§🎯 Core Features Summary

🔄 Auto-Persistence: Changes persist automatically via memory mapping
🔒 Zero Unsafe: Safe Rust APIs for all operations
⚡ Thread-Safe: Atomic fields work across processes
🎭 Convenience Operators: << for intuitive data manipulation
📏 Fixed-Size + Unlimited: Fixed layouts + segmented growth when needed
🧩 Rich Type System: Primitives, atomics, strings, collections, nested structs
📊 Multidimensional: N-dimensional arrays for join-like operations
🔌 Drop-in Compatible: All memmap2 code works unchanged

§🚀 Quick Start

Add to your Cargo.toml:

[dependencies]
memmap3 = "0.1"

Create persistent data structures:

use memmap3::prelude::*;

#[mmap_struct]
struct Counter {
    #[mmap(atomic)]
    value: u64,
    name: [u8; 32],
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create persistent data
    let mut counter = MmapStruct::<Counter>::create("/tmp/counter.mmap")?;
    &mut counter.name << "my_counter";
     
    // Atomic operations work across processes
    let old_value = counter.value.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
    println!("Counter: {} -> {}", old_value, old_value + 1);
     
    Ok(())
}

§🔧 The #`mmap_struct` Macro

Transform regular Rust structs into persistent, memory-mapped types:

use memmap3::prelude::*;

#[mmap_struct]
struct MyData {
    // Regular fields (primitives, enums)
    id: u64,
    active: bool,
     
    // Atomic fields (thread-safe across processes)
    #[mmap(atomic)]
    counter: u64,
     
    // Auto-detected strings (default for [u8; N])
    name: [u8; 32],
     
    // Explicit raw binary data
    #[mmap(raw)]
    key: [u8; 32],
     
    // Fixed-capacity vectors  
    #[mmap(vec)]
    scores: [u32; 10],
     
    // String arrays
    #[mmap(string_array)]
    tags: [[u8; 16]; 5],
     
    // Unlimited growth storage
    #[mmap(segmented)]
    events: [u64; 0],
}

§🎭 Convenience Operators

Smooth, intuitive syntax for common operations:

// String assignment
&mut my_string << "Hello";

// Vector append  
&mut my_vec << 42u32;

// HashMap insertion
&mut my_map << ("key".to_string(), "value".to_string());

// HashMap key assignment using macro
hashmap_index!(my_map["key2".to_string()] << "value2".to_string());

// HashMap access  
let value = &my_map[&"key".to_string()];

// Segmented append
&mut my_segmented << 123u64;

§🏗️ Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    memmap3 Safe Layer                       │
├─────────────────┬─────────────────┬─────────────────────────┤
│  #[mmap_struct] │   Collections   │     Atomic Types        │
│   ├─ Validation │   ├─ HashMap    │     ├─ MmapAtomicU64    │
│   ├─ Auto-Layout│   ├─ Vec        │     ├─ MmapAtomicI32    │
│   └─ Type Safety│   └─ String     │     └─ MmapAtomicBool   │
├─────────────────┴─────────────────┴─────────────────────────┤
│                    memmap2 Foundation                       │
│    ├─ Cross-platform memory mapping                         │
│    ├─ File lifecycle management                             │
│    └─ Low-level safety guarantees                           │
└─────────────────────────────────────────────────────────────┘

§📚 API Categories

§Core Types

MmapStruct<T> - Safe wrapper for memory-mapped structs
mmap_struct - Attribute macro for creating mappable structs

§Collections

MmapHashMap<K,V> - Persistent hash map with O(1) operations
MmapVec<T> - Persistent vector with dynamic growth
MmapString - Persistent UTF-8 string with auto-sizing

§Atomic Types

MmapAtomicU64, MmapAtomicI32, MmapAtomicBool - Thread-safe primitives
All standard atomic types with memory-mapped persistence

§Utilities

ByteArrayExt - String helpers for byte arrays
Segmented<T> - Advanced segmented storage for large datasets

§💡 Usage Patterns

§1. Simple Persistent Storage

use memmap3::prelude::*;

#[mmap_struct]
struct Config {
    max_connections: u32,
    timeout_ms: u32,
    debug_mode: bool,
    server_name: [u8; 64],
}

let mut config = MmapStruct::<Config>::create("/tmp/app.config")?;
config.max_connections = 1000;
config.timeout_ms = 5000;
config.debug_mode = false;
&mut config.server_name << "my-server";

// Values persist across application restarts
config.max_connections = 2000;

§2. Atomic Operations Across Processes

use memmap3::prelude::*;
use std::sync::atomic::Ordering;

#[mmap_struct]
struct Metrics {
    #[mmap(atomic)]
    requests_served: u64,
    #[mmap(atomic)]
    errors_count: u64,
    #[mmap(atomic)]
    last_restart: u64,
}

let metrics = MmapStruct::<Metrics>::create("/tmp/metrics.mmap")?;

// Thread-safe operations work across processes
metrics.requests_served.fetch_add(1, Ordering::SeqCst);

// Check if another process incremented the counter
let current = metrics.requests_served.load(Ordering::SeqCst);
println!("Total requests: {}", current);

§3. Persistent Collections

use memmap3::prelude::*;

// Persistent hash map
let mut cache = MmapHashMap::<&str, &str>::create("/tmp/cache.mmap")?;
cache.insert("user123", "Alice Smith")?;

// Persistent vector (fixed capacity)
let mut log = MmapVec::<u64, 1000>::new();
log.push(1234567890); // timestamp

// Persistent string (fixed size)
let mut name = MmapString::<64>::new();
&mut name << "Alice";

§4. String Operations

use memmap3::prelude::*;

#[mmap_struct]
struct UserInfo {
    id: u64,
    name: [u8; 64],    // Auto-detected as string
    email: [u8; 128],  // Auto-detected as string
}

let mut user = MmapStruct::<UserInfo>::create("/tmp/user.mmap")?;
user.id = 12345;

// String operations using << operator
&mut user.name << "Alice Smith";
&mut user.email << "alice@example.com";

// Read strings back
println!("User: {} ({})", user.name.as_str(), user.email.as_str());

§🔧 Feature Flags

serde - Enable Serde serialization support

memmap3 = { version = "0.1", features = ["serde"] }

§📖 Compare with Alternatives

Feature	memmap3	memmap2	mmap-rs	redb
Safe API	✅	❌	❌	✅
Zero-copy	✅	✅	✅	❌
Drop-in compatible	✅	N/A	❌	❌
Atomic operations	✅	❌	❌	❌
Cross-platform	✅	✅	✅	✅
Collections	✅	❌	❌	✅

§🎓 Learning Resources

Examples: See examples/ directory for comprehensive code samples
Tests: Integration tests in tests/ show real-world usage patterns
Benchmarks: Performance comparisons in benches/ directory

§⚠️ Safety Guarantees

memmap3 provides multiple layers of safety:

Compile-time validation through the #[mmap_struct] macro
Runtime bounds checking for all array and collection access
Memory layout verification ensuring struct compatibility
Atomic operation safety for concurrent access patterns
UTF-8 validation for all string operations

Note: While memmap3 eliminates unsafe code in user applications, memory-mapped files shared between processes require coordination to prevent data races. Use atomic operations for shared state.

Re-exports§

pub use types::ByteArrayExt;
pub use types::HashMapIterator;
pub use types::MmapAtomicBool;
pub use types::MmapAtomicI8;
pub use types::MmapAtomicI16;
pub use types::MmapAtomicI32;
pub use types::MmapAtomicI64;
pub use types::MmapAtomicIsize;
pub use types::MmapAtomicU8;
pub use types::MmapAtomicU16;
pub use types::MmapAtomicU32;
pub use types::MmapAtomicU64;
pub use types::MmapAtomicUsize;
pub use types::MmapHashMap;
pub use types::MmapString;
pub use types::MmapStringArray;
pub use types::MmapStruct;
pub use types::MmapVec;
pub use types::RecordMeta;
pub use types::Segmented;
pub use types::SegmentedIter;
pub use types::SegmentedMarker;
pub use types::SegmentedRecord;
pub use types::StructLayout;

Modules§

features: Optional features for memmap3
prelude: Prelude module for convenient imports
types: Safe memory-mapped data structures for persistent storage

Macros§

hashmap_index: Macro to enable my_map["key"] << "value" syntax

Structs§

Mmap: A handle to an immutable memory mapped buffer.
MmapMut: A handle to a mutable memory mapped buffer.
MmapOptions: A memory map builder, providing advanced options and flags for specifying memory map behavior.
MmapRaw: A handle to a raw memory mapped buffer.
MmapRawDescriptor
RemapOptions: Options for Mmap::remap and MmapMut::remap.

Enums§

Advice: Values supported by Mmap::advise and MmapMut::advise functions.
UncheckedAdvice: Values supported by Mmap::unchecked_advise and MmapMut::unchecked_advise functions.

Traits§

MmapAsRawDesc

Attribute Macros§

mmap_struct: Attribute macro for memory-mapped structs.

Crate memmap3

Crate memmap3 Copy item path

§🎯 Core Features Summary

§🚀 Quick Start

§🔧 The #mmap_struct Macro

§🎭 Convenience Operators

§🏗️ Architecture Overview

§📚 API Categories

§Core Types

§Collections

§Atomic Types

§Utilities

§💡 Usage Patterns

§1. Simple Persistent Storage

§2. Atomic Operations Across Processes

§3. Persistent Collections

§4. String Operations

§🔧 Feature Flags

§📖 Compare with Alternatives

§🎓 Learning Resources

§⚠️ Safety Guarantees

Re-exports§

Modules§

Macros§

Structs§

Enums§

Traits§

Attribute Macros§

Crate memmap3

§🔧 The #`mmap_struct` Macro