Crate flatdata

Implementation of heremaps/flatdata in Rust.

Flatdata is a library providing data structures for convenient creation, storage and access of packed memory-mappable structures with minimal overhead.

The idea is, that the user defines a schema of the data format using flatdata’s very simple schema language supporting plain structs, vectors and multivectors. The schema is then used to generate builders and readers for serialization and deserialization of the data. The data is serialized in a portable way which allows zero-overhead random access to it by using memory mapped storage. Memory mapped approach makes it possible to use the operating system facilities for loading, caching and paging of the data, and most important, accessing it as if it were in memory. Read more in “Why flatdata?”.

This crate provides:

• data structures for writing data to archives: StructBuf, Vector, ExternalVector, MultiVector
• data structures for reading data from archives: ArrayView, MultiArrayView
• resource storage backends for using archives: MemoryResourceStorage, FileResourceStorage, TarArchiveResourceStorage

The generator is part of the main heremaps/flatdata repository, the generate helper function is provided as a convenience wrapper.

For a comprehensive example, cf. coappearances schema and the corresponding usage.

Examples

First you design a schema for the data you want to store. Let’s say we want to store a list of prime factors for each natural number:

namespace prime {
// Represents a single prime factor of a number and how often it occurs.
struct Factor {
    value : u32 : 32;
    count : u32 : 8;
}

// Points towards the beginning of the list of prime numbers.
struct Number {
    @range(factors)
    first_factor_ref : u32;
}

// Stores a list of prime factors for numbers from 0 to N
archive Archive {
    @explicit_reference( Number.first_factor_ref, factors )
    numbers : vector<Number>;

    factors : vector<Factor>;
}
}

Maybe create a diagram using the dot generator from heremaps/flatdata:

Then you generate code using e.g. the generate utility in a build.rs script and include it in your project. Now you can create a (disk-based) archive and fill it with data:

include!("prime_generated.rs");

use flatdata::{ MemoryResourceStorage};

pub fn calculate_prime_factors(
    builder: &mut prime::ArchiveBuilder,
    max_number: u32,
) -> std::io::Result<()> {
    let mut numbers = builder.start_numbers()?;
    let mut factors = builder.start_factors()?;
    numbers.grow()?.set_first_factor_ref(0);
    for mut x in 0..=max_number {
        // Let's calculate prime factor in a very inefficient way
        for y in 2..x {
            let mut count = 0;
            while x % y == 0 {
                count += 1;
                x /= y;
            }
            if count > 0 {
                let mut factor = factors.grow()?;
                factor.set_value(y);
                factor.set_count(count);
            }
        }
        numbers.grow()?.set_first_factor_ref(factors.len() as u32);
    }
    numbers.close().expect("Failed to close");
    factors.close().expect("Failed to close");
    Ok(())
}

pub fn main() {
    let storage = MemoryResourceStorage::new("/primes");
let mut builder =
    prime::ArchiveBuilder::new(storage.clone()).expect("failed to create builder");
calculate_prime_factors(&mut builder, 10000).expect("Failed to write archive");
// store archive for re-use
// ...
// in a different application open archive for use:
let archive = prime::Archive::open(storage).expect("failed to open archive");
let number = 1234;
let factor_range = archive.numbers().at(number).first_factor_ref() as usize
    ..archive.numbers().at(number + 1).first_factor_ref() as usize;
let factors: Vec<_> = archive
    .factors()
    .slice(factor_range)
    .iter()
    .flat_map(|x| std::iter::repeat(x.value()).take(x.count() as usize))
    .collect();
println!("List if prime factors for {}: {:?}", number, factors);
}

This will print

List if prime factors for 1234 is [2, 617]

Optional Features

The following are a list of Cargo features that can be enabled or disabled:

• tar: Enables support for reading TAR archives using the TarArchiveResourceStorage struct.

Re-exports

• pub use crate::memory::PADDING_SIZE;
• pub use crate::storage::check_optional_resource;
• pub use crate::storage::check_resource;
• pub use crate::storage::create_archive;
• pub use crate::storage::create_external_vector;
• pub use crate::storage::create_multi_vector;

Structs

• ExternalVector
  Vector which flushes its content when growing.
• FileResourceStorage
  Resource storage on disk using memory mapped files.
• MemoryResourceStorage
  Resource storage in memory.
• MultiArrayView
  A read-only view on a multivector.
• MultiVector
  A container for writing an indexed sequence of heterogeneous data items.
• RawData
  Exposes blocks of raw data, providing auxiliary functionality like extracting substrings.
• TarArchiveResourceStorage
  Read-only resource storage on disk using a memory mapped tar archive.
• Vector
  A container holding a contiguous sequence of flatdata structs of the same type T in memory, and providing read and write access to it.

Enums

• GeneratorError
  Error type for generate function
• ResourceStorageError
  Error indicating failures when reading and writing data from/to a Storage.

Traits

• IndexStruct
  A specialized Struct factory producing Index items. Used primarily by the MultiVector/MultiArrayView.
• NoOverlap
  Marks structs that can be used stand-alone, e.g. no range
• Overlap
  Marks structs that cannot be used stand-alone, e.g. no range
• ResourceStorage
  Hierarchical Resource Storage
• SliceExt
  Enhanced slices of flatdata Structs so that they can be created from bytes / converted to bytes Note: TryFrom/AsRef cannot be used, since slice is a foreign type
• Struct
  A factory trait used to bind lifetime to Ref implementations.
• VariadicIndex
  A type used as element of ‘MultiArrayView’.
• VariadicRef
  A type used as element of MultiArrayView.
• VariadicRefFactory
  Shortcut trait for VariadicStructs that are able to produce references of any given lifetime
• VariadicStruct
  A type used to create VariadicStructs.

Functions

• generate
  A helper function wrapping the flatdata generator.

Type Aliases

• StorageHandle
  A handle to a resource storage used by archives
• TypeIndex
  Index specifying a variadic type of MultiArrayView.