Crate pigeonhole

Expand description

Embedded, single-file, wide-column store: BigTable’s data model with SQLite’s deployment model.

A Pigeonhole database is a sorted, sparse, versioned map (table, row, family, qualifier, timestamp) → value in one file.

use pigeonhole::{days, Durability, Family, Options, Pigeonhole};

let db = Pigeonhole::open(dir.join("crawl.phdb"), Options::default())?;
let pages = db
    .table("pages")?
    .family("meta", Family::default().max_versions(1))
    .family("links", Family::default().bloom_bits(10))
    .family("body", Family::default().blob_threshold(4096).ttl(days(30)))
    .family("stats", Family::counter())
    .create_if_missing()?;

// Single-row atomic mutation.
pages
    .mutate(b"com.example/a")
    .put("meta", b"status", b"200")
    .put("links", b"com.example/b", b"")
    .incr("stats", b"hits", 1)
    .delete_column("meta", b"etag")
    .commit()?;

// Point read: borrows from the cache, no allocation.
let status = pages.get(b"com.example/a", "meta", b"status")?;
assert_eq!(status.unwrap().value(), b"200");

// Row read, projected to families.
let row = pages.row(b"com.example/a").families(["meta", "stats"]).latest().read()?;
assert_eq!(row.unwrap().get("stats", b"hits").unwrap().as_i64(), Some(1));

// Ordered scan with filters pushed into the block decoder.
let snap = db.snapshot()?;
for row in pages
    .scan(b"com.example/"..b"com.example0")
    .family("links")
    .qualifier_prefix(b"org.")
    .snapshot(&snap)
    .iter()?
{
    let row = row?;
    assert!(row.is_empty() || row.key().starts_with(b"com.example/"));
}

// Batched multi-row write with one durability point.
let mut wb = db.write_batch();
wb.put(&pages, b"com.example/c", "meta", b"status", b"404");
wb.commit_with(Durability::GroupSync)?;
db.close()?;

§Install and status

cargo add pigeonhole, or pigeonhole = "0.1" in Cargo.toml. This is an experimental 0.x release: the core engine (Phase 1) is complete and fault-tested in simulation, but the on-disk format and the API may change before 1.0, the wide-column model (Phase 2) and the latency work (Phase 3) are still to come, and it is not recommended for production use yet. See the status page and the changelog.

§API shape and the future C ABI

Every zero-copy type (CellRef, RowRef) has an owned, cheap, ref-counted equivalent (Cell, Row), and every iterator has a cursor-style method (RowIter::next_ref), so a C ABI can wrap this crate later without exposing lifetimes, generics or closures. Errors are a flat, stable ErrorCode plus a message.

§Sync and async

Phase 1 ships the blocking API, which needs no async runtime. The async front door (Phase 3) lives behind the async feature in the nonblocking module.

Part of Pigeonhole. See the crate README.

Structs§

Cell: An owned cell version: a cheap ref-counted handle that pins its cache block. Send, Sync, 'static; safe to hold across .await.
CellEntry: One entry of a row: which column, and which version.
CellRef: A borrowed cell version, tied to the table handle that read it. Reading it allocates nothing; the bytes stay pinned until it drops. CellRef::to_owned keeps them longer.
CommitInfo: The outcome of a commit.
Error: An error: a stable ErrorCode and a human-readable message.
Family: A column family’s policy. Stored in the file with the family; a family’s kind and options are fixed when it is created.
MergeError: A merge operator failure (bad operand encoding, overflow policy).
Options: Database options. Process-local: nothing here is stored in the file, so reopening with different options changes them. Zero config is valid.
Pigeonhole: An open database: the one writer process’s handle. Cheap to clone; every clone shares the same engine and shard threads.
PigeonholeReader: A read-only handle, typically in another process. Has no write methods, so misuse does not compile. Reads run on the caller’s thread.
ReadTable: A table handle on a PigeonholeReader: reads only.
ReaderOptions: Options for a read-only handle in another process.
Row: An owned row. Cheap to clone.
RowIter: Rows of a scan, in key order. As an Iterator it yields owned Rows (cheap: values stay pinned, not copied); RowIter::next_ref lends zero-copy RowRefs instead.
RowMutation: A single-row atomic mutation under construction. Every change commits all-or-nothing, across families. Errors (unknown family, oversized key) surface at commit.
RowRead: A row read under construction. Finish with RowRead::read.
RowRef: A borrowed row: its key and its cells ordered by family, qualifier, newest version first.
Scan: An ordered scan under construction. Finish with Scan::iter.
Shard: One shard in application-owned mode. Move it to the thread that should run it and call Shard::run_once from that thread’s event loop.
Snapshot: A point-in-time view for reads. Holding it pins the data it can see; drop it promptly.
Table: A table handle for reads and writes. Cheap to clone; Send + Sync.
TableBuilder: Defines or opens a table. Returned by Pigeonhole::table.
Transaction: An optimistic multi-row transaction (Phase 4): serializable for the ranges it reads.
WriteBatch: A multi-row write with one durability point. Atomic across rows (two-phase commit when the rows live on different shards). Errors surface at commit.

Enums§

Compaction: Compaction strategy of a family.
Condition: A condition for a conditional mutation (RowMutation::commit_if).
Durability: How durable a commit must be before it returns. Ordered from weakest to strongest.
ErrorCode: A flat, stable error code. Values never change meaning and are never reused, so they map one-to-one onto a future C enum and onto exceptions in any binding.
Priority: Block-cache priority of a family.
Value: A typed view of a value.
ValueFilter: A predicate on a cell value, evaluated inside the read path before materialization.

Traits§

MergeOperator: A user-defined merge, applied at read and compaction time to operands written blind.

Functions§

days: n days, for TTLs: Family::default().ttl(days(30)). Saturates instead of overflowing.

Type Aliases§

Result: Result alias for this crate.