Crate serde_columnar

Crate serde_columnar 

Source
Expand description

§Introduction

serde_columnar is a crate that provides columnar storage for List and Map with compressible serialization and deserialization capabilities.

Columnar storage is very useful when you want to compress serialized data and you know that one or more fields of consecutive structs in the array have the same or equal difference values.

For example, you want to store this array:

[{a: 1, b: 1}, {a: 1, b: 2}, {a: 1, b: 3}, ...]

After columnar storage, it can be stored as:

a: [1, 1, 1,...] ---Rle---> [N, 1]
b: [1, 2, 3,...] ---DeltaRle---> [N, 1] (each value is 1 greater than the previous one)

§Usage

ⓘ
type ID = u64;
#[columnar(vec, ser, de)]
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct Data {
    #[columnar(strategy = "Rle")]
    num: u32,
    #[columnar(strategy = "DeltaRle", original_type = "u64")]
    id: ID,
    #[columnar(strategy = "Rle")]
    gender: String,
    #[columnar(strategy = "BoolRle")]
    married: bool
    #[columnar(strategy = "DeltaOfDelta")]
    time: i64
}

#[columnar]
#[derive(Debug, Serialize, Deserialize)]
pub struct VecStore {
    #[columnar(type = "vec")]
    pub data: Vec<Data>
}


let store = VecStore::new(...);
let bytes = serde_columnar::to_vec(&store).unwrap();
let store = serde_columnar::from_bytes::<VecStore>(&bytes).unwrap();

§More Details

§Container

  • #[columnar] means that some fields (marked by #[columnar(type = "vec"|"map")]) of this structure can be serialized and deserialized by columnar encoding
  • #[columnar(vec, map)] means the struct can be a row inside Vec-like or Map-like
  • #[columnar(ser, de)] means the struct can be serialized or deserialized or both by columnar encoding

§Field Attributes

  • #[columnar(type = "vec"|"map")]:
    • vec means the decorated field T is a container, holds Value and satisfies &T: IntoIter<Item=&Value> T: FromIterator<Value>
    • map means the decorated field T is a container, holds Value and satisfies &T: IntoIter<Item=(&K, &Value)> T: FromIterator<(K, Value)>
  • #[columnar(strategy = "Rle"|"BoolRle"|"DeltaRle"|"DeltaOfDelta")]: You can only choose one from
  • #[columnar(original_type="u32")]: this attribute is used to tell the columnar encoding the original type of the field, which is used when the field is a number
  • #[columnar(skip)]: the same as the skip attribute in serde

Modules§

__serde_utils
iterable

Macros§

flatten_tuple
izip
Create an iterator running multiple iterators in lockstep.
multi_zip

Structs§

AnyRleDecoder
AnyRleEncoder
BoolRleColumn
The Column that is scheduled to be compressed using BoolRleEncoder
BoolRleDecoder
BoolRleEncoder
ColumnAttr
ColumnarDecoder
The decoder of columnar system
ColumnarEncoder
The encoder of columnar system
ColumnarMap
The wrapper of Map-like container, we have implemented the Serialize and Deserialize for it.
ColumnarVec
The wrapper of Vec-like container, we have implemented the Serialize and Deserialize for it.
DeltaOfDeltaColumn
The Column that is scheduled to be compressed using [DeltaRleEncoder]
DeltaOfDeltaDecoder
DeltaOfDeltaEncoder
DeltaRleColumn
The Column that is scheduled to be compressed using DeltaRleEncoder
DeltaRleDecoder
DeltaRleEncoder
GenericColumn
RleColumn
The Column that is scheduled to be compressed using AnyRleEncoder

Enums§

ColumnarError
This is the error type used by serde_columnar
PostcardError
This is the error type used by Postcard

Traits§

ColumnTrait
DeltaRleable
Itertools
An Iterator blanket implementation that provides extra adaptors and methods.
KeyRowDe
The Map version of RowDe trait.
KeyRowSer
The Map version of RowSer trait.
MultiUnzip
An iterator that can be unzipped into multiple collections.
Rleable
RowDe
If a type implements RowSer and RowDe trait, it can be considered as a row of vec-like container.
RowSer
If a type implements RowSer and RowDe trait, it can be considered as a row of vec-like container.

Functions§

from_bytes
iter_from_bytes
to_vec

Attribute Macros§

columnar
Convenience macro to use the columnar system.