Crate swapvec

Source
Expand description

§SwapVec

A vector which swaps to disk when exceeding a certain length.

Useful if you do not want to use a queue, but first collecting all data and then consuming it.

Imagine multiple threads slowly producing giant vectors of data, passing it to a single consumer later on.

Or a CSV upload of multiple gigabytes to an HTTP server, in which you want to validate every line while uploading, without directly starting a Database transaction or keeping everything in memory.

§Features

  • Multiplatform (Linux, Windows, MacOS)
  • Creates temporary file only after exceeding threshold
  • Works on T: Serialize + Deserialize + Clone
  • Temporary file removed even when terminating the program
  • Checksums to guarantee integrity
  • Can be moved across threads

§Limitations

  • Due to potentially doing IO, most actions are wrapped in a Result
  • Currently, no “start swapping after n MiB” is implemented
    • Would need element wise space calculation due to heap elements (e.g. String)
  • Compression currently does not compress. It is there to keep the API stable.
  • No async support (yet)
  • When pushing elements or consuming iterators, SwapVec is “write only”
  • Only forwards iterations
    • Can be reset though

§Examples

§Basic Usage

use swapvec::SwapVec;
let iterator = (0..9).into_iter();
let mut much_data = SwapVec::default();
// Starts using disk for big iterators
much_data.consume(iterator).unwrap();
for value in much_data.into_iter() {
    println!("Read back: {}", value.unwrap());
}

§Examples

Currently there is only one simple example, doing some basic operations and getting metrics like getting the batches/bytes written to file. . Run it with

cargo run --example demo

Structs§

SwapVec
An only growing array type which swaps to disk, based on it’s initial configuration.
SwapVecConfig
Configure when and how the vector should swap.
SwapVecIter
Iterator for SwapVec.

Enums§

Compression
Configure compression for the temporary file into which your data might be swapped out.
CompressionLevel
Set compression level of the compression algorithm. This maps to different values depending on the chosen algortihm.
SwapVecError
A collection of all possible errors.

Traits§

Compress
Provide your own compression algorithm by creating an empty struct implementing compress and decompress.
CompressBoxedClone
Your custom compression algorithm struct must be debugable and clonable. Implement this trait to keep the main configuration debugable and clonable.