Struct shardio::ShardWriter

pub struct ShardWriter<T, S = DefaultSort> { /* fields omitted */ }

Write a stream data items of type T to disk, in the sort order defined by S.

Data is buffered up to item_buffer_size items, then sorted, block compressed and written to disk. When the ShardWriter is dropped or has finish() called on it, it will flush remaining items to disk, write an index of the chunk data and close the file.

The get_sender() methods returns a ShardSender that must be used to send items to the writer. You must close each ShardSender by dropping it or calling its finish() method, or data may be lost. The ShardSender must be dropped/finished prior to callinng SharWriter::finish or dropping the shard writer.

Sorting

Items are sorted according to the Ord implementation of type S::Key. Type S, implementing the SortKey trait maps items of type T to their sort key of type S::Key. By default the sort key is the data item itself, and the the DefaultSort implementation of SortKey is the identity function.

Methods

impl<T, S> ShardWriter<T, S> where
    T: 'static + Send + Serialize,
    S: SortKey<T>,
    <S as SortKey<T>>::Key: 'static + Send + Ord + Serialize + Clone, 

pub fn new<P: AsRef<Path>>(
    path: P,
    sender_buffer_size: usize,
    disk_chunk_size: usize,
    item_buffer_size: usize
) -> Result<ShardWriter<T, S>, Error>

Create a writer for storing data items of type T.

Arguments

• path - Path to newly created output file
• sender_buffer_size - number of items to buffer on the sending thread before transferring data to the writer
• disk_chunk_size - number of items to store in chunk on disk. Controls the tradeoff between indexing overhead and the granularity can be read at.
• item_buffer_size - number of items to buffer before writing data to disk. More buffering causes the data for a given interval to be spread over fewer disk chunks, but requires more memory.

pub fn get_sender(&self) -> ShardSender<T>

Get a ShardSender. It can be sent to another thread that is generating data.

pub fn finish(&mut self) -> Result<usize, Error>

Call finish if you want to detect errors in the writer IO.

Trait Implementations

impl<T, S> Drop for ShardWriter<T, S>

fn drop(&mut self)

Executes the destructor for this type.