Crate mokuroku

Expand description

Overview

This crate will build and maintain a secondary index within RocksDB, similar to what PouchDB does for LevelDB. The index keys and optional values are provided by your application. Once an index has been defined, queries for all entries, or those whose keys match a given prefix, can be performed. The index is kept up-to-date as data records, the data your application is storing in the database, are updated or deleted.

Usage

Create an instance of Database much like you would with rocksdb::DB. Provide the path to the database files, the set of indices to maintain (often referred to as views), and a ByteMapper that will assist in building indices from existing data.

Below is a very brief example. See the README and examples directory for additional examples.

fn mapper(key: &[u8], value: &[u8], view: &str, emitter: &Emitter) -> Result<(), Error> {
    // ... call emitter.emit() with index keys and optional values
    Ok(())
}
let db_path = "my_database";
let views = vec!["tags".to_owned()];
let dbase = Database::open_default(Path::new(db_path), views, Box::new(mapper)).unwrap();

Mutable

Due to a change in the Rust wrapper for RocksDB, the database reference must be mutable in order to make changes to the column families. This library makes many such changes, and as a result most of the methods require that the database reference is mutable.

Data Model

The secondary indices are built when Database.query() is called and the corresponding column family is missing. Knowing this, an application may want to open the database and subsequently call query() for every view. This will cause the index to be built based on the existing data. If for whatever reason the application deems it necessary to rebuild an index, that can be accomplished by calling the rebuild() function. When building an index, the library will invoke the application’s ByteMapper function.

N.B. The index key emitted by the application is combined with the data record primary key, separated by a single null byte. Using a separator is necessary since the library does not know the length or format of these keys in advance, and the index query depends heavily on using the prefix iterator to speed up the search. If you want to specify a different separator, use the Database.separator() function when opening the database. If at some later time you decide to change the separator, you will need to rebuild the indices.

Numeric Index Keys

To use numeric index keys, a reasonable approach is to use a value in Big-endian order and encode the value as base32hex using the functions in the base32 module provided in this crate. Doing so will allow for range queries on the numeric key, but keep in mind that the query keys must also be in Big-endian order and base32hex encoded.

Modules

base32

The base32 module provides encoding and decoding functions for converting binary data to and from UTF-8 alphanumeric text using the base32hex encoding defined in RFC-4648.