Expand description
This library provides backend and format agnostic data storage for simulation results coupled with metadata about the used Software and the simulation Run
The main use case is the following
- generate data on a cluster and save it with JSON backend
- transfer data to Neo4j backend
- directly use Neo4j to select data
Therefore the main goal is a simple solution for writing data and there are no plans to support advanched search or query features.
Data storage is not handled by the database, only associated metadata.
Currently two backends are included:
- Json, which saves everything in JSON files
- Neo4j, which uses a Neo4j database as backend (write only)
Custom backends can be implemented via the Database and DatabaseSession traits. Sessions are meant to associate a Datasets specific Run of a Software. Datasets are references to data stored in a file of any arbitrary format.
Features
json
enable Json backendneo4j
enable Neo4j backendsha
enable sha2 support for automatic hash calculationsarbitrary
enable support for arbitrary (required for tests)
Example
This creates a Json based Database and writes some arbitraty data to it. Note that in order to create a session, usually the vergen_session macro will suffice.
use std::io::Write;
use serde::Serialize;
use simuldb::prelude::*;
// Define a metadata type
#[derive(Debug, Serialize)]
struct Metadata {
a: usize,
b: String,
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Create or open database
let mut json = Json::new("output/json");
// Start new session which will contain references to the datasets
let software = Software::new("example", "1.0", "now");
let run = Run::new("now");
let session = Session::new(software, run);
let mut json_session = json.add_session(session)?;
// Create a directory for the result data
std::fs::create_dir_all("output/data")?;
// Generate some data and add it to the database
for a in 0_usize..10 {
// A DataWriter can be used to automatically calculate
// the hash of a file and create a Dataset from it
let mut writer = DatasetWriter::new("output/data")?;
// Write some data to the output file
writeln!(writer, "a^2 = {}", a.pow(2))?;
// Generate metadata to associate with it
let metadata = Metadata {
a,
b: "squaring".to_string(),
};
// Add the corresponding dataset to the database
let dataset = writer.finalize(metadata)?;
json_session.add_dataset(&dataset)?;
}
Ok(())
}
Re-exports
Modules
- Database backends
- Error types
- Commonly used structs, traits and macros
- Generic serialization target
Macros
- Generate session object automatically by extracting data from environment variables provided by Cargo and vergen
- Generate version for usage with vergen_session
Structs
- Reference to one data file
- Run metadata containing information about the machine and the start time
- Version information about the used Software
Functions
- Generate Map object from serializable metadata.