Crate snappy_framed [−] [src]
Normally, Snappy compression works entirely in memory, but
there is also a "Snappy framed" format which can be read and
written in streaming mode. We provide
implementations for framed snappy data.
The API to this library is designed to be similar to that of
flate2, though we have not yet implemented
The "Snappy framed" format includes CRC-32C checksums, which can be
computed relatively efficiently on Intel processors. However,
implementations disagree on the byte order of the checksums! The Java
snzip command-line tool and the
SmallTalk implementation use one byte order, and the Node.js and Python
implementations use another.
For now, we have chosen to generate checksums in the format used by the
snzip libraries. When reading, we can either verify
Java-format checksums, or we can ignore the checksums entirely.
This library is still a work in progress:
- Invalid streams will probably result in a panic.
- Decompression performance has been tuned a fair bit, except for CRCs, but there's probably an extra 25% or so to be gained by further tweaking.
- We currently assume that you will
writedata in large blocks when compressing, and we will generate poorly-compressed data if you make lots of small writes. This could be fixed by using an internal write buffer.