Crate serde_columnar

Introduction

serde_columnar is a crate that provides columnar storage for List and Map with compressible serialization and deserialization capabilities.

Columnar storage is very useful when you want to compress serialized data and you know that one or more fields of consecutive structs in the array have the same or equal difference values.

For example, you want to store this array:

[{a: 1, b: 1}, {a: 1, b: 2}, {a: 1, b: 3}, ...]

After columnar storage, it can be stored as:

a: [1, 1, 1,...] ---Rle---> [N, 1]
b: [1, 2, 3,...] ---DeltaRle---> [N, 1] (each value is 1 greater than the previous one)

Usage

type ID = u64;
#[columnar(vec, ser, de)]
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct Data {
    #[columnar(strategy = "Rle")]
    num: u32,
    #[columnar(strategy = "DeltaRle", original_type = "u64")]
    id: ID,
    #[columnar(strategy = "Rle")]
    gender: String,
    #[columnar(strategy = "BoolRle")]
    married: bool
}

#[columnar]
#[derive(Debug, Serialize, Deserialize)]
pub struct VecStore {
    #[columnar(type = "vec")]
    pub data: Vec<Data>
}


let store = VecStore::new(...);
let bytes = serde_columnar::to_vec(&store).unwrap();
let store = serde_columnar::from_bytes::<VecStore>(&bytes).unwrap();

More Details

Container

• #[columnar] means that some fields (marked by #[columnar(type = "vec"|"map")]) of this structure can be serialized and deserialized by columnar encoding
• #[columnar(vec, map)] means the struct can be a row inside Vec-like or Map-like
• #[columnar(ser, de)] means the struct can be serialized or deserialized or both by columnar encoding

Field Attributes

• #[columnar(type = "vec"|"map")]:
  • vec means the decorated field T is a container, holds Value and satisfies &T: IntoIter<Item=&Value> T: FromIterator<Value>
  • map means the decorated field T is a container, holds Value and satisfies &T: IntoIter<Item=(&K, &Value)> T: FromIterator<(K, Value)>
• #[columnar(strategy = "Rle"|"BoolRle"|"DeltaRle")]: You can only choose one from the three
  • Rle crate::strategy::AnyRleEncoder
  • BoolRle crate::strategy::BoolRleEncoder
  • DeltaRle crate::strategy::DeltaRleEncoder
• #[columnar(original_type="u32")]: this attribute is used to tell the columnar encoding the original type of the field, which is used when the field is a number
• #[columnar(skip)]: the same as the skip attribute in serde

Modules

• iterable

Macros

• flatten_tuple
• izip
  Create an iterator running multiple iterators in lockstep.
• multi_zip

Structs

• AnyRleDecoder
• AnyRleEncoder
• BoolRleColumn
  The Column that is scheduled to be compressed using BoolRleEncoder
• BoolRleDecoder
• BoolRleEncoder
• ColumnAttr
• ColumnDecoder
• ColumnEncoder
• ColumnarDecoder
  The decoder of columnar system
• ColumnarEncoder
  The encoder of columnar system
• ColumnarMap
  The wrapper of Map-like container, we have implemented the Serialize and Deserialize for it.
• ColumnarVec
  The wrapper of Vec-like container, we have implemented the Serialize and Deserialize for it.
• DeltaRleColumn
  The Column that is scheduled to be compressed using DeltaRleEncoder
• DeltaRleDecoder
• DeltaRleEncoder
• GenericColumn
• RleColumn
  The Column that is scheduled to be compressed using AnyRleEncoder

Enums

• ColumnarError
  This is the error type used by serde_columnar
• PostcardError
  This is the error type used by Postcard
• Strategy
  The enum of Strategy includes Rle/BoolRle/DeltaRle

Traits

• ColumnTrait
• DeltaRleable
• Itertools
  An Iterator blanket implementation that provides extra adaptors and methods.
• KeyRowDe
  The Map version of RowDe trait.
• KeyRowSer
  The Map version of RowSer trait.
• MultiUnzip
  An iterator that can be unzipped into multiple collections.
• Rleable
• RowDe
  If a type implements RowSer and RowDe trait, it can be considered as a row of vec-like container.
• RowSer
  If a type implements RowSer and RowDe trait, it can be considered as a row of vec-like container.

Functions

• from_bytes
• iter_from_bytes
• to_vec

Attribute Macros

• columnar
  Convenience macro to use the columnar system.