Crate swapvec

SwapVec

A vector which swaps to disk when exceeding a certain length.

Useful if you do not want to use a queue, but first collecting all data and then consuming it.

Imagine multiple threads slowly producing giant vectors of data, passing it to a single consumer later on.

Or a CSV upload of multiple gigabytes to an HTTP server, in which you want to validate every line while uploading, without directly starting a Database transaction or keeping everything in memory.

Features

• Multiplatform (Linux, Windows, MacOS)
• Creates temporary file only after exceeding threshold
• Works on T: Serialize + Deserialize + Clone
• Temporary file removed even when terminating the program
• Checksums to guarantee integrity
• Can be moved across threads

Limitations

• Due to potentially doing IO, most actions are wrapped in a Result
• Currently, no “start swapping after n MiB” is implemented
  • Would need element wise space calculation due to heap elements (e.g. String)
• Compression currently does not compress. It is there to keep the API stable.
• No async support (yet)
• When pushing elements or consuming iterators, SwapVec is “write only”
• Only forwards iterations
  • Can be reset though

Examples

Basic Usage

use swapvec::SwapVec;
let iterator = (0..9).into_iter();
let mut much_data = SwapVec::default();
// Starts using disk for big iterators
much_data.consume(iterator).unwrap();
for value in much_data.into_iter() {
    println!("Read back: {}", value.unwrap());
}

Examples

Currently there is only one simple example, doing some basic operations and getting metrics like getting the batches/bytes written to file. . Run it with

cargo run --example demo

Structs

SwapVec

An only growing array type which swaps to disk, based on it’s initial configuration.

SwapVecConfig

Configure when and how the vector should swap.

SwapVecIter

Iterator for SwapVec.

Enums

Compression

Configure compression for the temporary file into which your data might be swapped out.

CompressionLevel

Set compression level of the compression algorithm. This maps to different values depending on the chosen algortihm.

SwapVecError

A collection of all possible errors.

Traits

Compress

Provide your own compression algorithm by creating an empty struct implementing compress and decompress.

CompressBoxedClone

Your custom compression algorithm struct must be debugable and clonable. Implement this trait to keep the main configuration debugable and clonable.