Crate cql_db

This crate contains the core CQL Database functionality, orchestrating implementors of the CqlType trait allowing the system to act as an array-based database.

The library allows the consumers to provide a path to a local directory which will be used to store array based data as defined by the user. The number of dimensions in the array, and their maximum sizes must be stated on create of the database, however it will only allocate storage space for elements in the final (Nth) dimension upon linking of higher level dimensions.

Elements in the array can be writen to one by one, and read either as single points or to a stream.

Storage space consumption

This crate will allocate file space upon linking of dimensions, as well as a small amount on create of a database, so before starting you should be aware of the disk space requirements.

Given a database with N dimensions, calling create_db_unchecked will allocate (1 + N) * 8 bytes. Thereafter, linking a set of dimensions, will then expand the maximum file sizes according to the function below:

let database_definition = [6, 7, 8, 9, 10];
let link = [2, 3, 4, 5];

cql_db::link_dimensions_unchecked::<U64>(
    DATABASE_LOCATION,
    &link,
)?;

let mut key_file_size = 176; // total size of the key files in bytes

let n_dimensions_linked = 3; // +1 per key file
let n_elements_linked_between_second_and_third_dimension = 1; // includes this link
let n_elements_linked_between_third_and_fourth_dimension = 1; // includes this link

assert_eq!(
    (n_dimensions_linked +
        (
            (((link[0] - 1) * database_definition[1]) + link[1]) +
            (((n_elements_linked_between_second_and_third_dimension - 1) * database_definition[2]) + link[2]) +
            (((n_elements_linked_between_third_and_fourth_dimension - 1) * database_definition[3]) + link[3])
        )
    ) * 8,
    key_file_size
);

Should additional elements be linked, the key libraries will expand accordingly.

Additional space will be allocated for each penultimate dimenion (Nn-1) linked using the link_dimensions_unchecked function, this is equal to the maximum size of the final dimension multiplied by the VALUE_SIZE of the stored struct.

Examples

The following example creates a 4 dimensional database of unsigned 64 bit integers, links a chain of elements, writes a value, and then reads it:

use cql_u64::U64;

let point = [2, 4, 3, 1];
let value = 5;

// Create a database with a maximum capacity of `[2, 5, 3, 2]`
cql_db::create_db_unchecked::<U64>(
    DATABASE_LOCATION,
    &[2, 5, 3, 2]
)?;

// Link the 2nd element of the 1st dimension with the 4th element of the 2nd dimension, and
// the 4th of the 2nd with the 3rd of the 3rd - for example:
// Turbine 2 has data for Signal 4 for Year 3
cql_db::link_dimensions_unchecked::<U64>(
    DATABASE_LOCATION,
    &[2, 4, 3], // don't link the Nth dimension, can also be expressed as `&point[0..3]`
)?;

// Write value `value` to point `point`
cql_db::write_value_unchecked::<U64>(
    DATABASE_LOCATION,
    &point,
    value
)?;

// Read the stored value from point `point`
let result = cql_db::read_value_unchecked::<U64>(
    DATABASE_LOCATION,
    &point
)?;

assert_eq!(result, value);

Functions

create_db_unchecked	Creates an CQL database in the provided directory, overwriting existing files. Does not validate given parameters.
link_dimensions_unchecked	Links dimension indexs together if they are not already linked. Does not validate given parameters.
read_to_stream_unchecked	Reads n_values from the given location onward into the given stream. Does not validate given parameters.
read_value_unchecked	Reads the value at the given location from the database. Does not validate given parameters.
write_value_unchecked	Writes the given value to the given location in the database. Does not validate given parameters.