Crate rawdb 

rawdb

Non-transactional embedded storage engine with a filesystem-like API.

It features:

• Multiple named regions in one file
• Automatic space reclamation via hole punching
• Regions grow and move automatically as needed
• Zero-copy mmap access
• Thread-safe with concurrent reads and writes
• Page-aligned allocations (4KB)
• Crash-consistent with explicit flush
• Foundation for higher-level abstractions (e.g., vecdb)

It is not:

• A transactional database (no ACID, transactions, or rollback)
• A query engine (no SQL, indexes, or schemas)

Install

cargo add rawdb

Usage

use rawdb::{Database, Result};

fn main() -> Result<()> {
    // open database
    let temp_dir = tempfile::TempDir::new()?;
    let db = Database::open(temp_dir.path())?;

    // create regions
    let region1 = db.create_region_if_needed("region1")?;
    let region2 = db.create_region_if_needed("region2")?;

    // write data (buffered in mmap, not yet durable)
    region1.write(&[0, 1, 2, 3, 4])?;
    region2.write_at(&[5, 6, 7, 8, 9], 0)?;

    // flush to disk for durability
    db.flush()?;
    
    // read via mmap (data is immediately visible)
    {
        let reader = region1.create_reader();
        let _data = reader.read_all();
    } // reader dropped here, releasing its reference

    // remove region (space becomes reusable hole after flush)
    region1.remove()?;

    // punch holes in the file
    // db.compact()?; // doesn't work with doc-tests

    Ok(())
}

Durability

Operations become durable after calling flush(). Before flush, writes are visible in memory but not guaranteed to survive crashes.

Design:

• 4KB metadata entries: Atomic page-sized writes per region with embedded IDs
• Single metadata file: Rebuilt into HashMap on startup for O(1) lookups
• No WAL: Simple design with lazy flushing for consistency
• Lazy writes: Both data and metadata are written to mmaps immediately but not synced until flush

Write model:

1. Data writes modify the data mmap immediately (visible but not durable)
2. Metadata changes modify the metadata mmap immediately (visible but not durable)
3. Holes from moves/removes are marked as pending (not reusable until flush)
4. flush() syncs both mmaps (data → metadata → file size), then promotes pending holes
5. Ensures metadata never points to unflushed data and old locations aren’t reused prematurely

Region operations:

• Expand in-place when possible (last region or adjacent hole)
• Copy-on-write to new location when expansion needed
• All changes visible immediately in mmaps, durable after flush()

Recovery: On open, reads all metadata entries and rebuilds in-memory structures. Deleted regions are identified by zeroed metadata.

Re-exports

pub use error::*;

Modules

error

Structs

Database

Memory-mapped database with dynamic space allocation and hole punching.

DiskUsage

Represents actual disk usage of a file (accounting for sparse files and holes).

Reader

Zero-copy reader for accessing region data from memory-mapped storage.

Region

Named region within a database providing isolated storage space.

RegionMetadata

Metadata tracking a region’s location, size, and identity.

WeakDatabase

Weak reference to a Database that doesn’t prevent it from being dropped.

Constants

GiB

One gibibyte (1024^3 bytes).

PAGE_SIZE

PAGE_SIZE_MINUS_1

SIZE_OF_REGION_METADATA

Functions

likely

unlikely