Crate cql_db

This crate contains the core CQL Database functionality, orchestrating implementors of the CqlType trait allowing the system to act as an array-based database.

The library allows the consumers to provide a path to a local directory which will be used to store array based data as defined by the user. The number of dimensions in the array, and their maximum sizes must be stated on create of the database, however it will only allocate storage space for elements in the final (Nth) dimension upon linking of higher level dimensions.

Elements in the array can be writen to one by one, and read either as single points or to a stream.

Storage space consumption

This crate will allocate file space upon linking of dimensions, as well as a small amount on create of a database, so before starting you should be aware of the disk space requirements.

Given a database with N dimensions, calling create_db will allocate (1 + N) * 8 bytes. Thereafter, linking a set of dimensions, will then expand the maximum file sizes according to the function below:

let database_definition = [6, 7, 8, 9, 10];
let link = [2, 3, 4, 5];

cql_db::link_dimensions::<U64>(
    DATABASE_LOCATION,
    &link,
)?;

let mut key_file_size = 176; // total size of the key files in bytes

let n_dimensions_linked = 3; // +1 per key file
let n_elements_linked_between_second_and_third_dimension = 1; // includes this link
let n_elements_linked_between_third_and_fourth_dimension = 1; // includes this link

assert_eq!(
    (n_dimensions_linked +
        (
            (((link[0] - 1) * database_definition[1]) + link[1]) +
            (((n_elements_linked_between_second_and_third_dimension - 1) * database_definition[2]) + link[2]) +
            (((n_elements_linked_between_third_and_fourth_dimension - 1) * database_definition[3]) + link[3])
        )
    ) * 8,
    key_file_size
);

Should additional elements be linked, the key libraries will expand accordingly.

Additional space will be allocated for each penultimate dimenion (Nn-1) linked using the link_dimensions function, this is equal to the maximum size of the final dimension multiplied by the VALUE_SIZE of the stored struct.

Benchmarks

Benchmarks supplied below for the U64 type and are fairly rudimentary (and rounded) and are there to give a rough idea of relative costs. Full benchmark code can be found in github and can be run with rustup run nightly cargo bench. Benchmarks for other types can be found in the the type's corresponding documentation.

Operation	Database dimensions	Mean time _unchecked (ns)	Mean time (ns)
Single point read	1	2 450 (+/- 300)	7 500 (+/- 600)
Single point read	4	14 850 (+/- 1 000)	37 550 (+/- 2 300)
Single point write	1	2 800 (+/- 400)	7 700 (+/- 400)
Single point write	4	15 400 (+/- 2 500)	37 700 (+/- 3 000)
Stream read 1 point	1	2 500 (+/- 300)	10 000 (+/- 850)
Stream read 1 point	4	14 900 (+/- 600)	42 500 (+/- 6 500)
Stream read 50 000 points	1	27 650 000 (+/- 31 000)	27 630 000 (+/- 180 000)
Stream read 50 000 points	4	27 660 000 (+/- 1 200 000)	27 620 000 (+/- 480 000)

Examples

The following example creates a 4 dimensional database of unsigned 64 bit integers, links a chain of elements, writes a value, and then reads it:

use cql_u64::U64;

let point = [2, 4, 3, 1];
let value = 5;

// Create a database with a maximum capacity of `[2, 5, 3, 2]`
cql_db::create_db::<U64>(
    DATABASE_LOCATION,
    &[2, 5, 3, 2]
)?;

// Link the 2nd element of the 1st dimension with the 4th element of the 2nd dimension, and
// the 4th of the 2nd with the 3rd of the 3rd - for example:
// Turbine 2 has data for Signal 4 for Year 3
cql_db::link_dimensions::<U64>(
    DATABASE_LOCATION,
    &[2, 4, 3], // don't link the Nth dimension, can also be expressed as `&point[0..3]`
)?;

// Write value `value` to point `point`
cql_db::write_value::<U64>(
    DATABASE_LOCATION,
    &point,
    value
)?;

// Read the stored value from point `point`
let result = cql_db::read_value::<U64>(
    DATABASE_LOCATION,
    &point
)?;

assert_eq!(result, value);

Modules

error

Error types returned by cql_db

Functions

create_db	Creates an CQL database in the provided directory, if a database doesn't exist already.
create_db_unchecked	Creates an CQL database in the provided directory, overwriting existing files. Does not validate given parameters.
link_dimensions	Links dimension indexs together if they are not already linked.
link_dimensions_unchecked	Links dimension indexs together if they are not already linked. Does not validate given parameters.
read_to_stream	Reads n_values from the given location onward into the given stream.
read_to_stream_unchecked	Reads n_values from the given location onward into the given stream. Does not validate given parameters.
read_value	Reads the value at the given location from the database.
read_value_unchecked	Reads the value at the given location from the database. Does not validate given parameters.
write_value	Writes the given value to the given location in the database.
write_value_unchecked	Writes the given value to the given location in the database. Does not validate given parameters.