autumn-plugin-rocksdb

An Autumn plugin for RocksDB. Handlers read and write keys in an embedded store. The same database can hold the app cache and the sessions.

• Key-value API: get, put, delete, JSON values, column families and atomic batches.
• Scans: prefix and range scans in pages, with a cursor for the next page.
• Cache: RocksCache is the Autumn app cache. Each entry expires. Entries survive a restart.
• Sessions: RocksSessionStore is the Autumn session store. Sessions survive a restart.
• Limits: a timeout, a call limit, and size limits for keys, values, batches and scan pages.
• Operations: a readiness check, Prometheus metrics, checkpoints, and a flush and close at shutdown.
• Tests: use an in-memory database. The tests need no server.

Install

[dependencies]
autumn-plugin-rocksdb = "0.1"

use autumn_plugin_rocksdb::{RocksDb, RocksDbPlugin, RocksDbResultExt as _};
use autumn_web::prelude::*;

#[derive(serde::Deserialize, serde::Serialize)]
struct Profile {
    name: String,
}

#[get("/profiles/{id}")]
async fn profile(db: RocksDb, Path(id): Path<u64>) -> AutumnResult<Json<Profile>> {
    let profile = db
        .cf("profiles")
        .get_json::<Profile>(id.to_be_bytes())
        .await
        .or_http()?;
    profile.map(Json).ok_or_else(|| AutumnError::not_found_msg("no profile"))
}

#[autumn_web::main]
async fn main() {
    autumn_web::app()
        .plugin(RocksDbPlugin::new())
        .routes(routes![profile])
        .run()
        .await;
}

Add profiles to column_families in autumn.toml. See Configuration.

The first build compiles RocksDB from C++ source. It takes 10 to 15 minutes on 4 CPU cores. It needs a C++ compiler and clang.

The lz4, zstd and snappy features are on by default. They build these compression libraries into RocksDB. RocksDB writes Snappy blocks by default, so snappy lets the plugin read databases from other tools. The zlib feature adds zlib.

An app can have one RocksDB plugin only. Use column families to keep data sets apart.

The plugin re-exports the rocksdb crate and uses its types in with_db and setup hooks. A rocksdb update is a breaking release of this crate.

Configuration

The plugin reads [rocksdb] in autumn.toml. Profile sections and profile files override it. AUTUMN_ROCKSDB__<KEY> variables override all files. A list variable has comma-separated items.

[rocksdb]
path = "data/rocksdb"          # ":memory:" gives an in-memory database
access_mode = "read_write"     # or "read_only"
create_if_missing = true
column_families = ["profiles", "orders"]
timeout_ms = 5000              # includes the wait for a call slot
max_concurrent_calls = 64
max_key_bytes = 16384
max_value_bytes = 16777216
max_batch_bytes = 67108864
max_scan_entries = 1000
max_scan_bytes = 16777216
sync_writes = false            # true waits for the write-ahead log on disk
cache = false                  # true installs RocksCache as the app cache
cache_ttl_secs = 86400         # the longest cache TTL
sessions = false               # true installs RocksSessionStore
# session_ttl_secs = 86400     # default: session.max_age_secs of Autumn
health_check = true
flush_on_shutdown = true
compression = "lz4"            # "none", "lz4", "zstd" or "snappy"
bloom_filter_bits = 10         # 0 turns the bloom filter off
# block_cache_bytes = 67108864
# write_buffer_bytes = 67108864
# max_open_files = -1
# max_background_jobs = 4

The default path is :memory:. Set path to keep data after a restart.

On Unix, the plugin makes a new database directory with mode 0700. Only the owner can read it.

Use RocksDbPlugin::config to give a configuration in code. Use RocksDbPlugin::configure to change the configuration after the plugin reads it.

Reads and writes

The methods of RocksDb use the default column family. db.cf("name") gives a Keyspace for another column family. Add each column family to column_families.

db.put("user:1", "Ada").await?;
let name = db.get("user:1").await?;           // Option<Vec<u8>>
db.cf("profiles").put_json("user:1", &profile).await?;
db.delete("user:1").await?;

A batch writes all entries or none:

db.batch()
    .put("a", "1")
    .put_in("profiles", "b", "2")
    .delete("old")
    .commit()
    .await?;

A write that times out after it starts can still complete. Its outcome is unknown. A write that times out before it starts never runs.

Scans

A scan reads one page in key order. Page::next is the cursor of the next page. It is None when no entry is left.

let mut cursor = None;
loop {
    let mut scan = db.scan().prefix("user:").limit(100);
    if let Some(after) = cursor.take() {
        scan = scan.after(after);
    }
    let page = scan.fetch().await?;
    for entry in &page.entries {
        let profile: Profile = entry.value_as()?;
    }
    match page.next {
        Some(next) => cursor = Some(next),
        None => break,
    }
}

A page also ends at max_scan_bytes. One entry above that limit gives EntryTooLarge with the key of the entry. Read it with get, or give the key to after to skip it.

Cache and sessions

Set cache = true. The plugin installs RocksCache as the app cache. Autumn keeps one app cache for each process. #[cached] functions then store their values in RocksDB.

• Each cache entry expires. cache_ttl_secs is the longest TTL. An entry without a TTL gets it.
• A cache call does not wait. If all call slots are busy, a read is a miss and a write does nothing.
• A cache write does not wait for a RocksDB write stall. A stalled write does nothing.
• The cache has no size limit. Expired entries leave the disk at compaction.

Set sessions = true. The plugin installs RocksSessionStore. Each session expires session_ttl_secs after its last save. The key on disk is the SHA-256 hash of the session ID, not the ID.

A compaction filter removes expired cache entries and sessions from disk. A read after the expiry is a miss before the filter runs.

The cache and the sessions use the reserved column families autumn_cache and autumn_sessions. The user API refuses names with the autumn_ prefix.

Setup hooks and the full API

A setup hook runs at startup, before the first request. A failed hook stops the boot.

RocksDbPlugin::new().setup(|db| db.put(b"schema_version", b"1"))

with_db gives the full rocksdb API on a blocking thread, for example for a snapshot. The timeout and the call limit apply. sync_writes does not apply to writes in with_db or in setup hooks.

let value = db.with_db(|db| db.snapshot().get(b"key")).await?;

checkpoint makes a consistent copy of a file database, for example for a backup. The directory must not exist. On Unix, the copy has mode 0700. The copy holds all data, so keep it safe.

Errors

RocksDbError::status gives the HTTP status. or_http() converts a result for a handler.

The error text and the Debug output never show a RocksDB message, because it can hold a file path. detail() gives the full message. Do not show it to users. kind() gives the ErrorKind.

Operations

• Readiness: /actuator/health has a rocksdb component. It is down before startup and after the close. It is also down when writes stop, or when the background error count grew in the last 60 seconds.
• Metrics: rocksdb_calls_started_total, rocksdb_calls_total, rocksdb_calls_open, rocksdb_read_bytes_total, rocksdb_written_bytes_total, rocksdb_cache_requests_total and rocksdb_background_errors. Gauges for each column family give keys, file sizes, memtables and pending compaction.
• Shutdown: Autumn marks the shutdown before it drains the requests. At the mark, the plugin flushes the write-ahead log and the memtables. The database stays open for the drain.
• Close: after the drain, the plugin refuses new calls. It waits up to 5 seconds for open calls. Then it flushes a writable file database, if flush_on_shutdown is true. Then it closes the database.
• One process can open a database for writes. Other processes can open it with access_mode = "read_only". A read-only open is a view of the database at the time of the open. It does not see later writes.

License

Apache-2.0.