Crate snappy_framed [] [src]

Normally, Snappy compression works entirely in memory, but there is also a "Snappy framed" format which can be read and written in streaming mode. We provide Read and Write implementations for framed snappy data.

The API to this library is designed to be similar to that of flate2, though we have not yet implemented read::SnappyFramedEncoder or write::SnappyFramedDecoder.

A note about checksums

The "Snappy framed" format includes CRC-32C checksums, which can be computed relatively efficiently on Intel processors. However, implementations disagree on the byte order of the checksums! The Java implementations, the snzip command-line tool and the SmallTalk implementation use one byte order, and the Node.js and Python implementations use another.

For now, we have chosen to generate checksums in the format used by the Java and snzip libraries. When reading, we can either verify Java-format checksums, or we can ignore the checksums entirely.

Limitations

This library is still a work in progress:

  • Invalid streams will probably result in a panic.
  • Decompression performance has been tuned a fair bit, except for CRCs, but there's probably an extra 25% or so to be gained by further tweaking.
  • We currently assume that you will write data in large blocks when compressing, and we will generate poorly-compressed data if you make lots of small writes. This could be fixed by using an internal write buffer.

Modules

read

Implementations of Read using Snappy compression.

write

Implementations of Write using Snappy compression.