Crate simuldb

This library provides backend and format agnostic data storage for simulation results coupled with metadata about the used Software and the simulation Run

The main use case is the following

1. generate data on a cluster and save it with JSON backend
2. transfer data to Neo4j backend
3. directly use Neo4j to select data

Therefore the main goal is a simple solution for writing data and there are no plans to support advanched search or query features.

Data storage is not handled by the database, only associated metadata.

Currently two backends are included:

• Json, which saves everything in JSON files
• Neo4j, which uses a Neo4j database as backend (write only)

Custom backends can be implemented via the Database and DatabaseSession traits. Sessions are meant to associate a Datasets specific Run of a Software. Datasets are references to data stored in a file of any arbitrary format.

Features

• json enable Json backend
• neo4j enable Neo4j backend
• sha enable sha2 support for automatic hash calculations
• arbitrary enable support for arbitrary (required for tests)

Example

This creates a Json based Database and writes some arbitraty data to it. Note that in order to create a session, usually the vergen_session macro will suffice.

use std::io::Write;
use serde::Serialize;
use simuldb::prelude::*;

// Define a metadata type
#[derive(Debug, Serialize)]
struct Metadata {
    a: usize,
    b: String,
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create or open database
    let mut json = Json::new("output/json");

    // Start new session which will contain references to the datasets
    let software = Software::new("example", "1.0", "now");
    let run = Run::new("now");
    let session = Session::new(software, run);
    let mut json_session = json.add_session(session)?;

    // Create a directory for the result data
    std::fs::create_dir_all("output/data")?;

    // Generate some data and add it to the database
    for a in 0_usize..10 {
        // A DataWriter can be used to automatically calculate
        // the hash of a file and create a Dataset from it
        let mut writer = DatasetWriter::new("output/data")?;

        // Write some data to the output file
        writeln!(writer, "a^2 = {}", a.pow(2))?;

        // Generate metadata to associate with it
        let metadata = Metadata {
            a,
            b: "squaring".to_string(),
        };

        // Add the corresponding dataset to the database
        let dataset = writer.finalize(metadata)?;
        json_session.add_dataset(&dataset)?;
    }

    Ok(())
}

Re-exports

pub use chrono;

pub use uuid;

pub use neo4rs;

Modules

db

Database backends

error

Error types

prelude

Commonly used structs, traits and macros

value

Generic serialization target

Macros

vergen_session

Generate session object automatically by extracting data from environment variables provided by Cargo and vergen

vergen_version

Generate version for usage with vergen_session

Structs

Dataset

Reference to one data file

Host

Run

Run metadata containing information about the machine and the start time

Session

Combination of Software and Run metadata associated with the data of one run

Software

Version information about the used Software

Functions

serialize_metadata

Generate Map object from serializable metadata.