mappedpages

A crash-consistent, memory-mapped, file-backed fixed-size page provider for Rust.

mappedpages manages a binary file divided into fixed-size pages, addressable by PageId. It is intended as a low-level building block for higher-level allocators and storage systems.

Features

• Compile-time page-size safety — Pager, PageId, and ProtectedPageId are all generic over a const PAGE_SIZE: usize. A PageId<1024> cannot be passed to a Pager<4096> — the compiler rejects the mismatch.
• Crash consistency — every allocation state change is committed via a double-buffered write: the inactive metadata page is written and synced first, then the superblock pointer is flipped and synced. The active metadata page is never overwritten in place.
• Protected pages — copy-on-write pages backed by two physical pages. Writes are staged in the inactive copy and atomically promoted on commit, surviving any crash between the two.
• Borrow-checked safety — &MappedPage and &mut MappedPage hold a borrow on the Pager that produced them. alloc and free both require &mut Pager, so the borrow checker statically prevents accessing a page reference after a remap — a compile error, not a runtime hazard.
• Dynamic growth — the file grows automatically when space is exhausted, with safe recovery if a remap fails mid-grow.
• CRC32 checksums — every metadata page and directory block is protected by a CRC32 checksum. On open, the library validates both copies and falls back to the alternate if one is corrupt.
• Sub-page allocation — SubPageAllocator divides big pages into smaller uniform sub-pages, so callers don't have to implement their own bitmap-based slab allocators on top of the raw pages. Implements BulkPageAllocator for sub-pages, so alloc_bulk and free_bulk work there too.
• Async I/O support — async versions of allocation and deallocation methods are available with the "async" feature flag, enabling integration with async runtimes like Tokio.
• Bulk operations — alloc_bulk(count) and free_bulk(ids) allocate or free multiple regular pages in a single crash-safe metadata commit. alloc_protected_bulk and free_protected_bulk provide the same convenience for protected pages. SubPageAllocator also supports alloc_bulk/free_bulk. All free_bulk variants validate all ids atomically so no partial state change occurs on error. The BulkPageAllocator trait lets generic code require this capability with a single where-bound.
• Page iterators — iter_allocated_pages() returns an AllocatedPageIter over regular data pages (internal protected-page resources are excluded). iter_allocated_protected_pages() returns an AllocatedProtectedPageIter over in-use protected pages. The two iterators are strictly disjoint: neither leaks the other's pages.

File layout

Page 0  — Superblock   (magic, version, page size, active metadata selector + checksum)
Page 1  — Metadata A   (free bitmap, total pages, generation, checksum)
Page 2  — Metadata B   (same layout; alternate buffer for crash-safe commits)
Page 3+ — Data pages   (user-visible, returned by alloc)

The minimum page size is 1024 bytes; violating this or using a non-power-of-two PAGE_SIZE is a compile error.

The page size is a permanent property of the file. It is written to the superblock on create and validated against PAGE_SIZE on every subsequent open. All allocation — including sub-page allocation — happens within the data bytes of pages that already have this fixed size on disk.

Installation

Add to your Cargo.toml:

[dependencies]
mappedpages = "0.1"

Usage

Creating and opening a pager

The page size is specified as a const generic on Pager. It must be a power of two and at least 1024 — both constraints are enforced at compile time.

use mappedpages::Pager;

// Create a new file with 4096-byte pages.
let mut pager = Pager::<4096>::create("data.bin")?;

// Open an existing file — returns InvalidPageSize if the on-disk size differs.
let mut pager = Pager::<4096>::open("data.bin")?;

Allocating and accessing pages

PageId<PAGE_SIZE> is an opaque handle tied to the same page size as the Pager that produced it. Using a handle with the wrong pager is a compile error.

use mappedpages::{Pager, PageId};

let mut pager = Pager::<4096>::create("data.bin")?;

// Allocate a new page.
let id: PageId<4096> = pager.alloc()?;

// Write to the page — requires &mut Pager.
{
    let page = id.get_mut(&mut pager)?;
    page.as_bytes_mut().fill(0xAB);
} // mutable borrow released here

// Read from the page — requires &Pager.
{
    let page = id.get(&pager)?;
    println!("{:?}", &page.as_bytes()[..4]);
}

// Free the page when done.
pager.free(id)?;

Protected (crash-consistent) pages

Protected pages use copy-on-write: a write is staged in the inactive physical copy and only becomes visible after an explicit commit. Dropping a ProtectedPageWriter without committing discards the write.

use mappedpages::{Pager, ProtectedPageId};

let mut pager = Pager::<4096>::create("data.bin")?;

// Allocate a protected page.
let id: ProtectedPageId<4096> = pager.alloc_protected()?;

// Stage a write.
{
    let mut writer = id.get_mut(&mut pager)?;
    writer.page_mut().as_bytes_mut().fill(0xFF);
    writer.commit()?; // atomically makes the write durable
}

// Read the active copy.
{
    let page = id.get(&pager)?;
    assert_eq!(page.as_bytes()[0], 0xFF);
}

PageAllocator and BulkPageAllocator traits

PageId, ProtectedPageId, and SubPageId all implement PageHandle for their respective allocator types. Pager<N> implements PageAllocator for both PageId and ProtectedPageId; SubPageAllocator implements it for SubPageId. This allows generic code to work with any page type:

use mappedpages::{PageAllocator, PageHandle, Pager, PageId};

fn fill_pages<H, const N: usize>(pager: &mut Pager<N>, n: usize) -> Vec<H>
where
    H: PageHandle<Pager<N>>,
    Pager<N>: PageAllocator<H>,
{
    (0..n).map(|_| pager.alloc().unwrap()).collect()
}

BulkPageAllocator<H> is a supertrait of PageAllocator<H> for allocators that support efficient or all-or-nothing batch operations. It is implemented by Pager<N> for both PageId and ProtectedPageId, and by SubPageAllocator for SubPageId. Requiring it is as simple as adding a bound:

use mappedpages::{BulkPageAllocator, PageHandle};

fn load_bulk<A, H>(allocator: &mut A, n: usize) -> Vec<H>
where
    H: PageHandle<A>,
    A: BulkPageAllocator<H>,
{
    allocator.alloc_bulk(n).unwrap()
}

Sub-page allocation

The file format always uses a single fixed page size — the size is written to the superblock on create and checked on every open. When your workload needs smaller granularity than the on-disk page size, SubPageAllocator handles the bookkeeping for you so you don't have to build your own bitmap-based slab allocator on top of raw pages.

SubPageAllocator<PARENT_SIZE, SUB_SIZE> wraps a Pager<PARENT_SIZE>, divides each big page it checks out into PARENT_SIZE / SUB_SIZE sub-slots, and exposes them as SubPageId handles that plug into the same PageHandle / PageAllocator traits. It also implements BulkPageAllocator, so alloc_bulk and free_bulk are available for sub-pages with the same all-or-nothing validation semantics. Up to 64 sub-pages per big page are supported.

Important: sub-allocation state is in-memory only. The on-disk file is unchanged — sub-slots live inside the data bytes of normal pages — but the free/used bitmasks are not written to disk. On process restart, reconstruct your sub-allocation state from your own records before using sub-page handles again.

use mappedpages::{PageAllocator, PageHandle, Pager, SubPageAllocator};

// Wrap a pager; no big pages are checked out yet.
let pager = Pager::<4096>::create("data.bin")?;
let mut sub = SubPageAllocator::<4096, 512>::new(pager);

// Each sub-page is 512 bytes; 8 fit in one 4096-byte big page.
let id = sub.alloc()?;                                   // SubPageId<4096, 512>
sub.alloc_mut(id)?.as_bytes_mut().fill(0xAB);           // write 512 bytes
assert_eq!(id.get(&sub)?.len(), 512);

sub.free(id)?;   // when all 8 sub-slots in a big page are freed,
                 // the big page is returned to the inner pager

Recover the inner pager when you are done:

let pager: Pager<4096> = sub.into_pager();

Async I/O Support

Async versions of allocation and deallocation methods are available when the "async" feature is enabled. These methods allow integration with async runtimes like Tokio.

Add to your Cargo.toml:

[dependencies]
mappedpages = { version = "0.1", features = ["async"] }
tokio = { version = "1", features = ["full"] }

Usage:

use mappedpages::Pager;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut pager = Pager::<4096>::create("data.bin")?;

    // Allocate asynchronously
    let id = pager.alloc_async().await?;

    // Use the page (synchronous access)
    {
        let page = id.get_mut(&mut pager)?;
        page.as_bytes_mut().fill(0xAB);
    }

    // Free asynchronously
    pager.free_async(id).await?;

    Ok(())
}

Note: Currently, the async methods block the async runtime thread due to underlying memory map flush operations. Future versions may provide truly non-blocking async I/O.

Bulk operations

alloc_bulk and free_bulk perform multiple regular-page allocations or deallocations with only a single crash-safe metadata commit at the end, reducing overhead compared to individual alloc/free calls. free_bulk validates all ids atomically before touching the bitmap — a single invalid id returns an error and leaves all pages unchanged.

use mappedpages::Pager;

let mut pager = Pager::<4096>::create("data.bin")?;

// Allocate 10 pages in one commit.
let ids = pager.alloc_bulk(10)?;
assert_eq!(ids.len(), 10);

// Free all 10 pages in one commit.
pager.free_bulk(ids)?;

alloc_protected_bulk and free_protected_bulk work the same way for protected pages. Because each protected-page allocation involves multiple physical pages and directory commits, the operation cannot be batched into a single commit; it does guarantee that on failure all already-allocated protected pages are freed (alloc) and that all ids are validated before any page is freed (free).

let pids = pager.alloc_protected_bulk(5)?;
pager.free_protected_bulk(pids)?;

Async variants are available under the "async" feature flag:

let ids = pager.alloc_bulk_async(10).await?;
pager.free_bulk_async(ids).await?;

let pids = pager.alloc_protected_bulk_async(5).await?;
pager.free_protected_bulk_async(pids).await?;

Iterating allocated pages

iter_allocated_pages traverses the allocation bitmap and yields a PageId for each allocated regular data page. Internal protected-page resources (directory block pages and backing pages for in-use protected entries) are excluded. Reserved pages 0–2 are never included. The iterator holds an immutable borrow on the pager.

iter_allocated_protected_pages traverses the protected-page directory and yields a ProtectedPageId for each in-use slot. Regular data pages are never included. The two iterators are strictly disjoint.

use mappedpages::Pager;

let mut pager = Pager::<4096>::create("data.bin")?;
let _reg  = pager.alloc_bulk(3)?;
let _prot = pager.alloc_protected_bulk(2)?;

// Iterate regular pages only.
for id in pager.iter_allocated_pages() {
    let page = id.get(&pager)?;
    println!("regular page {}: first byte = {}", id.0, page.as_bytes()[0]);
}

// Iterate protected pages only.
for pid in pager.iter_allocated_protected_pages() {
    let page = pid.get(&pager)?;
    println!("protected page {}: first byte = {}", pid.0, page.as_bytes()[0]);
}

API

Pager<PAGE_SIZE>

The central type. All page handles hold a borrow on the Pager that produced them.

Method	Signature	Description
create	(path) -> Result<Self>	Create a new file; fails if it already exists
open	(path) -> Result<Self>	Open and validate an existing file; fails if the on-disk page size ≠ PAGE_SIZE
alloc	(&mut self) -> Result<PageId<PAGE_SIZE>>	Allocate a regular page; grows the file if needed
free	(&mut self, PageId<PAGE_SIZE>) -> Result<()>	Free a regular page
alloc_bulk	(&mut self, usize) -> Result<Vec<PageId<PAGE_SIZE>>>	Allocate n regular pages in one commit; grows as needed
free_bulk	(&mut self, Vec<PageId<PAGE_SIZE>>) -> Result<()>	Free multiple regular pages in one commit; validates all ids atomically
iter_allocated_pages	(&self) -> AllocatedPageIter<'_, PAGE_SIZE>	Iterator over allocated regular data pages (excludes internal protected resources)
iter_allocated_protected_pages	(&self) -> AllocatedProtectedPageIter<'_, PAGE_SIZE>	Iterator over in-use protected pages
alloc_async	(&mut self) -> Result<PageId<PAGE_SIZE>>	Async version of alloc (requires "async" feature)
free_async	(&mut self, PageId<PAGE_SIZE>) -> Result<()>	Async version of free (requires "async" feature)
alloc_bulk_async	(&mut self, usize) -> Result<Vec<PageId<PAGE_SIZE>>>	Async version of alloc_bulk (requires "async" feature)
free_bulk_async	(&mut self, Vec<PageId<PAGE_SIZE>>) -> Result<()>	Async version of free_bulk (requires "async" feature)
alloc_protected	(&mut self) -> Result<ProtectedPageId<PAGE_SIZE>>	Allocate a crash-consistent copy-on-write page
free_protected	(&mut self, ProtectedPageId<PAGE_SIZE>) -> Result<()>	Free a protected page and both its backing copies
alloc_protected_bulk	(&mut self, usize) -> Result<Vec<ProtectedPageId<PAGE_SIZE>>>	Allocate n protected pages; rolls back on failure
free_protected_bulk	(&mut self, Vec<ProtectedPageId<PAGE_SIZE>>) -> Result<()>	Free multiple protected pages; validates all ids before freeing any
alloc_protected_async	(&mut self) -> Result<ProtectedPageId<PAGE_SIZE>>	Async version of alloc_protected (requires "async" feature)
free_protected_async	(&mut self, ProtectedPageId<PAGE_SIZE>) -> Result<()>	Async version of free_protected (requires "async" feature)
alloc_protected_bulk_async	(&mut self, usize) -> Result<Vec<ProtectedPageId<PAGE_SIZE>>>	Async version of alloc_protected_bulk (requires "async" feature)
free_protected_bulk_async	(&mut self, Vec<ProtectedPageId<PAGE_SIZE>>) -> Result<()>	Async version of free_protected_bulk (requires "async" feature)
page_size	(&self) -> usize	Page size in bytes (always equal to PAGE_SIZE)
page_count	(&self) -> u64	Total pages in the file, including reserved pages 0–2
free_page_count	(&self) -> u64	Pages currently available for allocation

AllocatedPageIter<'_, PAGE_SIZE>

An iterator over allocated regular data pages, yielded by Pager::iter_allocated_pages. Implements Iterator<Item = PageId<PAGE_SIZE>>. Reserved pages 0–2 and all internal protected-page resources (directory block pages and backing pages for in-use protected entries) are never yielded. Holds an immutable borrow on the Pager for its lifetime.

AllocatedProtectedPageIter<'_, PAGE_SIZE>

An iterator over in-use protected pages, yielded by Pager::iter_allocated_protected_pages. Implements Iterator<Item = ProtectedPageId<PAGE_SIZE>>. Regular data pages and internal directory-block pages are never included. Holds an immutable borrow on the Pager for its lifetime.

PageId<PAGE_SIZE>

Opaque handle to a regular data page. Cheap to copy. Can only be used with a Pager<PAGE_SIZE> of the same size.

Method	Signature	Description
get	(&self, &'a Pager<PAGE_SIZE>) -> Result<&'a MappedPage>	Immutably borrow the page
get_mut	(&self, &'a mut Pager<PAGE_SIZE>) -> Result<&'a mut MappedPage>	Mutably borrow the page

MappedPage

Unsized view into one page of the memory map (analogous to str or Path). Always held behind a reference.

Method	Signature	Description
as_bytes	(&self) -> &[u8]	Raw byte slice of the page
as_bytes_mut	(&mut self) -> &mut [u8]	Mutable raw byte slice
len	(&self) -> usize	Page size in bytes
is_empty	(&self) -> bool	Always false for valid pages

ProtectedPageId<PAGE_SIZE>

Opaque handle to a crash-consistent copy-on-write page. Cheap to copy. Can only be used with a Pager<PAGE_SIZE> of the same size.

Method	Signature	Description
get	(&self, &'a Pager<PAGE_SIZE>) -> Result<&'a MappedPage>	Read the active copy
get_mut	(&self, &'a mut Pager<PAGE_SIZE>) -> Result<ProtectedPageWriter<'a, PAGE_SIZE>>	Begin a staged write

ProtectedPageWriter<'_, PAGE_SIZE>

In-progress write to a protected page. Dropping without commit leaves the active copy unchanged.

Method	Signature	Description
page_mut	(&mut self) -> &mut MappedPage	Mutable view of the page being written
commit	(self) -> Result<()>	Flush and atomically promote the write to active

SubPageAllocator<PARENT_SIZE, SUB_SIZE>

Divides big pages from a Pager<PARENT_SIZE> into sub-pages of SUB_SIZE bytes. Owns the inner pager. Sub-allocation state is in-memory only — not persisted and not crash-consistent.

Compile-time constraints: SUB_SIZE must be a power of two, PARENT_SIZE must be divisible by SUB_SIZE, SUB_SIZE < PARENT_SIZE, and PARENT_SIZE / SUB_SIZE ≤ 64.

Method	Signature	Description
new	(Pager<PARENT_SIZE>) -> Self	Wrap a pager; no big pages are checked out until the first alloc
alloc	(&mut self) -> Result<SubPageId<PARENT_SIZE, SUB_SIZE>>	Allocate one sub-page; checks out a new big page from the inner pager when needed
free	(&mut self, SubPageId<...>) -> Result<()>	Free a sub-page; returns the big page to the inner pager when all its sub-slots are free
alloc_bulk	(&mut self, usize) -> Result<Vec<SubPageId<...>>>	Allocate n sub-pages; rolls back already-allocated sub-pages on failure
free_bulk	(&mut self, Vec<SubPageId<...>>) -> Result<()>	Free multiple sub-pages; validates all ids atomically, releases any fully-freed big pages
pager	(&self) -> &Pager<PARENT_SIZE>	Borrow the inner pager (e.g. to query page_count)
into_pager	(self) -> Pager<PARENT_SIZE>	Consume this allocator and recover the inner pager

SubPageId<PARENT_SIZE, SUB_SIZE>

Opaque handle to one allocated sub-page. Cheap to copy. Can only be used with a SubPageAllocator of matching PARENT_SIZE and SUB_SIZE.

Method	Signature	Description
get	(&self, &'a SubPageAllocator<...>) -> Result<&'a MappedPage>	Immutably borrow the sub-page (len() == SUB_SIZE)
get_mut	(&self, &'a mut SubPageAllocator<...>) -> Result<&'a mut MappedPage>	Mutably borrow the sub-page

PageAllocator / BulkPageAllocator / PageHandle traits

PageHandle<A> is implemented by PageId<N>, ProtectedPageId<N>, and SubPageId<P, S> for their respective allocator types. PageAllocator<H> is implemented by Pager<N> for the first two and by SubPageAllocator<P, S> for the third. BulkPageAllocator<H> is an optional supertrait of PageAllocator<H> that adds alloc_bulk and free_bulk; it is implemented by Pager<N> for both PageId and ProtectedPageId, and by SubPageAllocator<P, S> for SubPageId. Generic code that requires bulk capability adds a where A: BulkPageAllocator<H> bound.

Error handling

All fallible operations return Result<_, MappedPageError>. Notable variants:

Variant	Meaning
InvalidPageSize	On-disk page size does not match PAGE_SIZE (open); non-power-of-two or < 1024 is a compile error (create)
CorruptSuperblock	Unrecognised magic or file too small
CorruptMetadata	Both metadata copies failed checksum
OutOfBounds	PageId refers to a non-existent page
DoubleFree	Freeing an already-free page
Unavailable	Pager is unusable after a failed remap; reopen the file