RockSolid Store
RockSolid Store is a Rust library providing a robust, ergonomic, and opinionated persistence layer on top of the powerful RocksDB embedded key-value store. It aims to simplify common database interactions, focusing on ease of use, Column Family (CF) support, automatic serialization, transactions, batching, merge operations, and key expiration hints, while offering flexibility for advanced use cases.
Features ✨
- Column Family (CF) Aware:
RocksDbCFStore: The primary, public store for fine-grained non-transactional operations on any Column Family.RocksDbStore: A convenience wrapper aroundRocksDbCFStorefor simple, non-batch, non-transactional operations on the default Column Family.RocksDbCFTxnStore: The primary, public store for CF-aware transactional operations usingrocksdb::TransactionDB.RocksDbTxnStore: A convenience wrapper aroundRocksDbCFTxnStorefor transactional operations primarily focused on the default Column Family.
- Flexible Configuration:
RocksDbCFStoreConfig/RocksDbCFTxnStoreConfig: Comprehensive configuration for CF-aware stores, allowing per-CF tuning profiles, merge operators, and comparators.RocksDbStoreConfig/RocksDbTxnStoreConfig: Simplified configuration for default-CF focused usage.TuningProfile: Pre-defined RocksDB option sets for common workloads (DB-wide or per-CF) via thetunermodule (e.g.,LatestValue,MemorySaver,RealTime,TimeSeries,SparseBitmap). Each profile can include I/O capping options.RockSolidComparatorOpt: Specify custom key comparison strategies (e.g., natural sorting) per Column Family (requires "natlex_sort" or "nat_sort" features).
- Simplified API: Clear methods for common operations (e.g.,
get,put,merge,delete, iterators) targeting specific CFs via theCFOperationstrait implemented byRocksDbCFStoreandRocksDbCFTxnStore(for committed reads). - Automatic Serialization: Seamlessly serialize/deserialize keys (using
bytevec) and values (using MessagePack viarmp_serde) with minimal boilerplate viaserde. - Value Expiry Hints: Built-in support for associating expiry timestamps (Unix epoch seconds) with values using
ValueWithExpiry<T>. (Note: Automatic data removal based on expiry (e.g., TTL compaction) requires further RocksDB configuration not directly managed by RockSolid's basic API; application logic or custom compaction filters are typically needed). - Transactional Support (CF-Aware):
RocksDbCFTxnStore: Provides fully CF-aware pessimistic transactional operations. Obtainrocksdb::Transactionviabegin_transaction()and use methods likeput_in_txn_cf(),get_in_txn().RocksDbTxnStore: Offers aTransactionContextfor managing operations primarily on the default CF, andexecute_transaction()for automatic commit/rollback on the default CF.
- Atomic Batch Writes (CF-Aware): Efficiently group multiple write operations into atomic batches targeting specific CFs using the fluent
BatchWriterAPI (obtained fromRocksDbCFStore::batch_writer(cf_name)orRocksDbStore::batch_writer()). Multi-CF atomic batches are possible usingraw_batch_mut(). Non-transactional. - Configurable Merge Operators (Per-CF): Supports standard merge operators and includes a flexible Merge Router (
MergeRouterBuilder) to dispatch merge operations based on key patterns. Configurable per Column Family.- ⚠️ Merge Router Warning: The router functions created by
MergeRouterBuilderuse globally shared static routers. If you use a merge operator name derived fromMergeRouterBuilderacross different Column Families (even in different database instances within the same process), they will share the same routing rules. Ensure your key patterns or key prefixes are unique across these CFs to prevent unintended routing conflicts.
- ⚠️ Merge Router Warning: The router functions created by
- Error Handling: Uses a clear
StoreErrorenum (includingUnknownCf) andStoreResult<T>. - Utilities (CF-Aware): Includes utilities for database backup (checkpointing) and migration (
backup_db,migrate_db) aware of Column Families for non-transactional stores (usingRocksDbCFStoreConfig). - Helper Macros: Optional macros (
generate_dao_*andgenerate_dao_*_cf) to reduce boilerplate for common data access patterns on default-CF or specific-CF stores.
Installation ⚙️
Add rocksolid to your Cargo.toml:
[]
= "2.1.0" # Ensure this matches the latest version
= { = "1.0", = ["derive"] }
To use custom comparators like natural sorting, enable the corresponding features:
[]
= { = "2.1.0", = ["natlex_sort", "nat_sort"] }
= { = "1.0", = ["derive"] }
RockSolid relies on the rocksdb crate. Ensure you have the necessary system dependencies for rocksdb (like clang, libclang-dev, llvm-dev). See the rust-rocksdb documentation for details.
Quick Start: Default Column Family (Non-Transactional) 🚀
For simple use cases focusing on the default Column Family (non-batch, non-transactional operations):
use ;
use ;
use tempdir;
Quick Start: Column Families (Non-Transactional) 🚀
For applications requiring multiple Column Families or explicit CF control (non-transactional):
use ;
use ;
use ;
use StoreResult;
use ;
use HashMap;
use tempdir;
const PRODUCTS_CF: &str = "products_cf";
const INVENTORY_CF: &str = "inventory_cf";
Transactional Usage 🏦
RockSolid provides CF-aware transactional stores.
1. Default CF Focused Transactions (using RocksDbTxnStore)
use ;
use DefaultCFOperations; // For store.put, store.get on committed state
use ;
use tempdir;
2. CF-Aware Transactions (using RocksDbCFTxnStore)
use ;
use BaseCfConfig;
use StoreResult;
use CFOperations; // For store.get_in_txn, store.put_in_txn_cf
use ;
use HashMap;
use tempdir;
const ORDERS_CF: &str = "orders_cf";
const CUSTOMERS_CF: &str = "customers_cf";
Merge Operators & Routing (Per-CF) 🔄
Configure merge operators per-CF via BaseCfConfig in RocksDbCFStoreConfig (or CFTxConfig for transactional stores).
use ;
use ;
use ;
use StoreResult;
use MergeOperands; // Important for merge function signature
use ;
use ;
use tempdir;
const SETS_CF: &str = "sets_cf";
;
// Example merge function. Operands for custom merge operators are raw byte slices.
// The application logic defines how these bytes are interpreted (e.g., as serialized MergeValue<SimpleSet>).
Warning on MergeRouterBuilder: The router functions created by MergeRouterBuilder (i.e., router_full_merge_fn, router_partial_merge_fn) use globally shared static routers. This means if you register a merge operator name (e.g., "MyRouter") derived from a MergeRouterBuilder, all Column Families (even across different database instances in the same process) that use this "MyRouter" name will share the same routing rules. If your route patterns (e.g., /lists/{id}) are generic, ensure your actual keys within different CFs are sufficiently unique (e.g., prefixed like /cf1_lists/{id}, /cf2_lists/{id}) to avoid unintended routing conflicts if those CFs share the same routed merge operator.
Key/Value Expiry Hints ⏱️ (CF-Aware)
Store values with an expiry timestamp in any CF using RocksDbCFStore or the default CF with RocksDbStore.
use ;
use RocksDbCFStoreConfig;
use ValueWithExpiry;
use StoreResult;
use ;
use tempdir;
use ;
use HashMap;
const SESSIONS_CF: &str = "sessions_cf";
Configuration Structures ⚙️
RocksDbCFStoreConfig: ForRocksDbCFStore(non-transactional). Allows DB-wide and per-CF tuning, merge operators, and comparators. UsesBaseCfConfigfor per-CF settings.RocksDbStoreConfig: ForRocksDbStore(non-transactional). Simplified for default CF, with options for default CF's tuning, merge operator, comparator, plus DB-wide hard settings.RocksDbCFTxnStoreConfig: ForRocksDbCFTxnStore(transactional). Similar toRocksDbCFStoreConfigbut usesCFTxConfig(which wrapsBaseCfConfig) and includesTransactionDBOptions.RocksDbTxnStoreConfig: ForRocksDbTxnStore(transactional). Simplified for default CF, includesTransactionDBOptions. (Note: direct comparator setting is not on this struct; it's applied via conversion toBaseCfConfigif needed, or through custom options).BaseCfConfig: Per-CF settings (tuning_profile,merge_operator: Option<RockSolidMergeOperatorCfConfig>,comparator: Option<RockSolidComparatorOpt>) used within non-transactional CF configs and wrapped byCFTxConfig.CFTxConfig: Per-CF settings for transactional stores, wrapsBaseCfConfig.TuningProfile: (Fromrocksolid::tuner) Enum for pre-defined option sets.RockSolidComparatorOpt: (Fromrocksolid::config) Enum to specify custom key comparison (e.g.,NaturalLexicographical,Natural). Requires "natlex_sort" or "nat_sort" features.RockSolidMergeOperatorCfConfigvsMergeOperatorConfig:RockSolidMergeOperatorCfConfigis used byRocksDbCFStoreConfig,RocksDbStoreConfig, andBaseCfConfig. The olderMergeOperatorConfig(re-exported for compatibility) is used byRocksDbTxnStoreConfigfor itsdefault_cf_merge_operator.
Refer to src/config.rs, src/tx/cf_tx_store.rs, and src/tx/tx_store.rs for detailed structures.
API Reference 📚
For a detailed list of functions and types, please refer to the generated rustdoc documentation.
Key components and their CF-aware nature:
- Stores:
RocksDbCFStore: Primary non-transactional CF-aware store. ImplementsCFOperations.- Methods like
get(cf_name, ...),put(cf_name, ...),batch_writer(cf_name).
- Methods like
RocksDbStore: Wrapper for non-transactional default CF. ImplementsDefaultCFOperations.- Methods like
get,put(on default CF). AccessesRocksDbCFStoreviacf_store().batch_writer()for default CF.
- Methods like
RocksDbCFTxnStore: Primary transactional CF-aware store. ImplementsCFOperations(for committed reads).- Methods like
get(cf_name, ...)(committed read),put_in_txn_cf(&txn, cf_name, ...),begin_transaction().
- Methods like
RocksDbTxnStore: Wrapper for transactional default CF focus. ImplementsDefaultCFOperations(for committed reads/writes).- Methods like
get,put(auto-commit on default CF),transaction_context().
- Methods like
CFOperationsTrait: Defines the interface for CF-aware data operations.DefaultCFOperationsTrait: Defines the interface for default CF data operations.BatchWriter(fromRocksDbCFStore::batch_writer(cf_name)orRocksDbStore::batch_writer()): Non-transactional. CF-bound on creation for high-level methods. Methods likeset,delete, etc., operate on its bound CF. Useraw_batch_mut()for multi-CF atomicity.TransactionContext(fromRocksDbTxnStore::transaction_context()): Manages a transaction on the default CF.Tx<'a>(fromstore.begin_transaction()): Rawrocksdb::Transactionobject.- Configuration: As detailed above.
- Macros:
- Default CF macros (e.g.,
generate_dao_get!) work with stores implementingDefaultCFOperations. - CF-Aware macros (e.g.,
generate_dao_get_cf!) work with stores implementingCFOperationsand take acf_name. - Transactional macros (e.g.,
generate_dao_merge_in_txn!,generate_dao_remove_in_txn!) are available. Note thatgenerate_dao_put_in_txn!andgenerate_dao_put_with_expiry_in_txn!currently rely on static helper functions that are not defined in therocksolid::txmodule; for these operations, useTransactionContextor directTxmethods.
- Default CF macros (e.g.,
- Utilities (
utils.rs):backup_db,migrate_dbuseRocksDbCFStoreConfigand are CF-aware for non-transactional stores.
Running Examples
Example paths are relative to xs_rs/rocksolid/.
Contributing 🤝
Contributions are welcome!
Focus areas:
- Enhanced examples, especially for advanced tuning and
TransactionDBOptions. - Expanding
TuningProfiles with more specific workloads. - Integration of RocksDB's TTL compaction filters with
ValueWithExpiry(could be per-CF). - Performance benchmarks.
License 📜
This project is licensed under the Mozilla Public License Version 2.0.