Struct flatdata::MultiVector

pub struct MultiVector<Idx, Ts> { /* fields omitted */ }

A container for writing an indexed sequence of heterogeneous data items.

The concept of a multivector is used for storing and reading heterogeneous flatdata structs in/from the same container. The data is indexed by integers. Each index refers to a bucket which may contain a variable number of items of different types unified in the same variant enum Ts. Such bucket may also be empty, which allows to represent sparse data in a multivector. For those who are familiar with C++'s std::multimap data structure, a multivector can be considered as a std::multimap mapping integers to sequences of variable length.

A MultiVector corresponds rather to ExternalVector than to Vector, in the sense that the items are flushed to storage whenever the internal buffer is full. In particular, it is only possible to modify the last bucket. There is no access to the buckets previously stored.

For accessing and reading the data stored by in multivector, cf. MultiArrayView.

A multivector must be closed, after the last element was written to it. After closing, it can not be used anymore. Not closing the multivector will result in panic on drop (in debug mode).

Internally data is stored like this:

• Index: Vector<Idx> - encodes start/end byte in Data array for each element i. * Data: Vec<u8> - sequence of serialized (Tag, ItemData) tuples, where Tag encodes the the variant type, and ItemData contains the underlying variant data. Tag has size of 1 byte, ItemData is of size Ts::Type::SIZE_IN_BYTES.

Examples

use flatdata::{
    create_multi_vector, ArrayView, MultiArrayView, MultiVector,
    Struct, MemoryResourceStorage, ResourceStorage,
};

// define structs usually generated by flatdata's generator

define_index!(
    Idx, IdxMut, "some_idx_schema", 4, 32
);

define_struct!(A, AMut, "some_A_schema", 4,
    (x, set_x, u32, 0, 16),
    (y, set_y, u32, 16, 16)
);

define_struct!(B, BMut, "some_B_schema", 2,
    (id, set_id, u32, 0, 16)
);

define_variadic_struct!(AB, ABItemBuilder, Idx,
    0 => (A, add_a),
    1 => (B, add_b)
);

// create multivector and serialize some data

let mut storage = MemoryResourceStorage::new("/root/multivec".into());
{
    let mut mv = create_multi_vector::<Idx, AB>(
            &mut storage, "multivector", "some schema")
        .expect("failed to create MultiVector");
    {
        let mut item = mv.grow().expect("grow failed");
        {
            let mut a = item.add_a();
            a.set_x(1);
            a.set_y(2);
        }
        {
            let mut b = item.add_b();
            b.set_id(42);
        }
    }
    mv.close().expect("close failed");
}

// open index and data, and read the data

let index_resource = storage
    .read_and_check_schema("multivector_index", "index(some schema)")
    .expect("read_and_check_schema failed");
let index: ArrayView<Idx> = ArrayView::new(&index_resource);
let resource = storage
    .read_and_check_schema("multivector", "some schema")
    .expect("read_and_check_schema failed");
let mv: MultiArrayView<Idx, AB> = MultiArrayView::new(index, &resource);

assert_eq!(mv.len(), 1);
let mut item = mv.at(0);
let a = item.next().unwrap();
match *a {
    AB::A(ref a) => {
        assert_eq!(a.x(), 1);
        assert_eq!(a.y(), 2);
    },
    _ => assert!(false),
}
let b = item.next().unwrap();
match *b {
    AB::B(ref b) => {
        assert_eq!(b.id(), 42);
    },
    _ => assert!(false),
}

Methods

impl<Idx: Index, Ts: VariadicStruct> MultiVector<Idx, Ts>

pub fn new(index: ExternalVector<Idx>, data_handle: ResourceHandle) -> Self

Creates an empty multivector.

pub fn grow(&mut self) -> Result<HandleMut<Ts::ItemBuilder>>

Appends a new item to the end of this multivector and returns a builder for it.

The builder is used for storing different variants of Ts in the newly created item.

Calling this method may flush data to storage (cf. flush), which may fail due to different IO reasons.

pub fn close(&mut self) -> Result<()>

Flushes the remaining not yet flushed elements in this multivector and finalizes the data inside the storage.

After this method is called, more data cannot be written into this multivector. A multivector must be closed, otherwise it will panic on drop (in debug mode).

Trait Implementations

impl<Idx, Ts> Drop for MultiVector<Idx, Ts>

fn drop(&mut self)

Executes the destructor for this type.

impl<Idx: Index, Ts: VariadicStruct> Debug for MultiVector<Idx, Ts>

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.