fjall 3.0.0-pre.0

Log-structured, embeddable key-value storage engine
Documentation

[!WARNING] This is the 3.0.0 source code - the 2.0.0 source is available at https://github.com/fjall-rs/fjall/tree/2.0.0

Fjall (Nordic: "Mountain") is a log-structured, embeddable key-value storage engine written in Rust. It features:

  • A thread-safe BTreeMap-like API
  • 100% safe & stable Rust
  • LSM-tree-based storage similar to RocksDB
  • Range & prefix searching with forward and reverse iteration
  • Multiple keyspaces (a.k.a. column families) with cross-keyspace atomic semantics
  • Built-in compression (default = LZ4)
  • Serializable transactions (optional)
  • Key-value separation for large blob use cases (optional)
  • Automatic background maintenance

It is not:

  • A standalone database server
  • A relational or wide-column database: it has no built-in notion of columns or query language

Sponsors

Basic usage

cargo add fjall

use fjall::{PersistMode, Database, KeyspaceCreateOptions};

// A database may contain multiple keyspaces
// You should probably only use a single database for your application
let db = Database::builder(folder).open()?;
// TxDatabase::builder for transactional semantics

// Each keyspace is its own physical LSM-tree, and thus isolated from other keyspaces
let items = db.keyspace("my_items", KeyspaceCreateOptions::default())?;

// Write some data
items.insert("a", "hello")?;

// And retrieve it
let bytes = items.get("a")?;

// Or remove it again
items.remove("a")?;

// Search by prefix
for kv in items.prefix("prefix") {
  // ...
}

// Search by range
for kv in items.range("a"..="z") {
  // ...
}

// Iterators implement DoubleEndedIterator, so you can search backwards, too!
for kv in items.prefix("prefix").rev() {
  // ...
}

// Sync the journal to disk to make sure data is definitely durable
// When the database is dropped, it will try to persist with `PersistMode::SyncAll` automatically
keyspace.persist(PersistMode::SyncAll)?;

[!TIP] Like any typical key-value store, keys are stored in lexicographic order. If you are storing integer keys (e.g. timeseries data), you should use the big endian form to have predicatable ordering.

Durability

To support different kinds of workloads, Fjall is agnostic about the type of durability your application needs. After writing data (insert, remove or committing a write batch/transaction), you can choose to call Database::persist which takes a PersistMode parameter. By default, any operation will flush to OS buffers, but not to disk. This is in line with RocksDB's default durability. Also, when dropped, the database will try to persist the journal to disk synchronously.

Multithreading, Async and Multiprocess

[!WARNING] A single database may not be loaded in parallel from separate processes.

Fjall is internally synchronized for multi-threaded access, so you can clone around the Database and Keyspaces as needed, without needing to lock yourself.

For an async example, see the tokio example.

Feature flags

lz4

Allows using LZ4 compression, powered by lz4_flex.

Enabled by default.

single_writer_tx

Allows opening a transactional database for single-writer (serialized) transactions, allowing RYOW (read-your-own-write), fetch-and-update and other atomic operations.

Enabled by default.

ssi_tx

Allows opening a transactional database for multi-writer, serializable transactions, allowing RYOW (read-your-own-write), fetch-and-update and other atomic operations. Conflict checking is done using optimistic concurrency control.

Disabled by default. Mutually exclusive with single_writer_tx.

bytes

Uses bytes as the underlying Slice type.

Disabled by default.

Stable disk format

Future breaking changes will result in a major version bump and a migration path.

For the underlying LSM-tree implementation, see: https://crates.io/crates/lsm-tree.

Examples

See here for practical examples.

And checkout Smoltable, a standalone Bigtable-inspired toy wide-column database using fjall as its storage engine.

Contributing

How can you help?

License

All source code is licensed under MIT OR Apache-2.0.

All contributions are to be licensed as MIT OR Apache-2.0.