farena

A file-backed arena allocator.

Write data to a temporary file, then read it back by location. The data stays on disk instead of in memory, so your process doesn't use extra RAM.

Use this when you need scratch space for bytes but can't afford to keep everything in memory.

Temp files are created in your system's temp directory (respects TMPDIR).

Make sure your temp directory is on real disk, not in memory. Check with:

df -h ${TMPDIR:-/tmp}

The filesystem should not be tmpfs or ramfs.

Usage

use farena::{FileArena, FileArenaWriter, Location};

// Write phase
let mut writer = FileArenaWriter::new()?;
let (offset1, len1) = writer.push("hello")?;
let (offset2, len2) = writer.push(" world")?;
let file = writer.finish()?.unwrap();

// Read phase
let arena = FileArena::new(vec![file])?;
let loc1 = Location::new(0, offset1, len1);
let loc2 = Location::new(0, offset2, len2);

assert_eq!(arena.get(loc1)?, b"hello");
assert_eq!(arena.get(loc2)?, b" world");

Multiple files

let mut w1 = FileArenaWriter::new()?;
let (o1, l1) = w1.push("data1")?;
let f1 = w1.finish()?.unwrap();

let mut w2 = FileArenaWriter::new()?;
let (o2, l2) = w2.push("data2")?;
let f2 = w2.finish()?.unwrap();

let arena = FileArena::new(vec![f1, f2])?;
let loc1 = Location::new(0, o1, l1);  // file index 0
let loc2 = Location::new(1, o2, l2);  // file index 1

Buffer reuse

let mut buf = Vec::new();
arena.get_into(loc1, &mut buf)?;  // appends to buf