Crate swapvec

SwapVec

A vector which swaps to disk when exceeding a certain length.

Useful if you do not want to use a queue, but first collecting all data and then consuming it.

Imagine multiple threads slowly producing giant vectors of data, passing it to a single consumer later on.

Or a CSV upload of multiple gigabytes to an HTTP server, in which you want to validate every line while uploading, without directly starting a Database transaction or keeping everything in memory.

Features

• Multiplatform (Linux, Windows, MacOS)
• Creates temporary file only after exceeding threshold
• Works on T: Serialize + Deserialize + Clone
• Temporary file removed even when terminating the program
• Checksums to guarantee integrity
• Can be moved across threads

Limitations

• Due to potentially doing IO, most actions are wrapped in a Result
• Currently, no “start swapping after n MiB” is implemented
  • Would need element wise space calculation due to heap elements (e.g. String)
• Compression currently does not compress. It is there to keep the API stable.
• No async support (yet)
• When pushing elements or consuming iterators, SwapVec is “write only”
• Only forwards iterations
  • Can be reset though

Examples

Basic Usage

use swapvec::SwapVec;
let iterator = (0..9).into_iter();
let mut much_data = SwapVec::default();
// Starts using disk for big iterators
much_data.consume(iterator).unwrap();
for value in much_data.into_iter() {
    println!("Read back: {}", value.unwrap());
}

Examples

Currently there is only one simple example, doing some basic operations and getting metrics like getting the batches/bytes written to file. . Run it with

cargo run --example demo

Structs

• SwapVec
  An only growing array type which swaps to disk, based on it’s initial configuration.
• SwapVecConfig
  Configure when and how the vector should swap.
• SwapVecIter
  Iterator for SwapVec.

Enums

• Compression
  Configure compression for the temporary file into which your data might be swapped out.
• CompressionLevel
  Set compression level of the compression algorithm. This maps to different values depending on the chosen algortihm.
• SwapVecError
  A collection of all possible errors.

Traits

• Compress
  Provide your own compression algorithm by creating an empty struct implementing compress and decompress.
• CompressBoxedClone
  Your custom compression algorithm struct must be debugable and clonable. Implement this trait to keep the main configuration debugable and clonable.