seqdb

A K.I.S.S. (Keep It Simple, Stupid) sequential storage engine that provides memory-mapped file-based storage with dynamic region management.

What is seqdb?

seqdb is a lightweight storage engine designed for applications that need to store and retrieve data in named regions on disk. It provides:

Memory-mapped file access for fast I/O operations
Dynamic region management with automatic resizing and defragmentation
Sequential writes optimized for append-heavy workloads
Hole punching to reclaim unused disk space
Thread-safe operations using parking_lot locks

Key Features

Named regions: Store data in logical regions identified by strings or numbers
Automatic space management: Regions grow dynamically and can be moved/defragmented
Memory efficiency: Uses memory mapping for zero-copy reads
Cross-platform hole punching: Supports Linux, macOS, and FreeBSD
Thread-safe: Concurrent access using RwLocks

Usage

use std::{fs, path::Path};
use seqdb::{Database, PAGE_SIZE, Result};

fn main() -> Result<()> {
    // Create or open a database
    let database = Database::open(Path::new("my_db"))?;

    // Create a region
    let (region_id, _) = database.create_region_if_needed("my_region")?;

    // Write data to the region
    database.write_all_to_region(region_id.into(), b"Hello, world!")?;

    // Write at a specific offset
    database.write_all_to_region_at(region_id.into(), b"Hi", 0)?;

    // Read data using a reader
    let reader = database.create_region_reader(region_id.into())?;
    let data = reader.read_all();

    // Truncate region to specific length
    database.truncate_region(region_id.into(), 5)?;

    // Flush changes and reclaim space
    database.flush_then_punch()?;

    Ok(())
}

Core Types

Database: Main entry point for database operations
Identifier: Region identifier (string or number)
Reader: Zero-copy reader for region data
Region: Metadata about storage regions

Storage Model

seqdb organizes data into regions within a single memory-mapped file. Each region has:

Start offset: Position in the file (page-aligned)
Length: Current data size
Reserved space: Allocated space (≥ length, page-aligned)

Regions can grow automatically and are moved/defragmented as needed for efficient space utilization.