brotli-decompressor 2.2.0

A brotli decompressor that with an interface avoiding the rust stdlib. This makes it suitable for embedded devices and kernels. It is designed with a pluggable allocator so that the standard lib's allocator may be employed. The default build also includes a stdlib allocator and stream interface. Disable this with --features=no-stdlib. Alternatively, --features=unsafe turns off array bounds checks and memory initialization but provides a safe interface for the caller. Without adding the --features=unsafe argument, all included code is safe. For compression in addition to this library, download https://github.com/dropbox/rust-brotli
Documentation

rust-brotli-decompressor

crates.io Build Status

What's new in version 2.2.x

  • into_impl for reader and writer classes
  • removed BrotliStateCleanup since it happens upon drop()

What's new in version 2.1.2

  • Better handling of transient errors ( fixes #4 )
  • Do not panic in debug mode on unexpected bytes (handle arithmetic overflow)
  • Smaller stack allocations
  • Never create slice::from_raw_parts with nil
  • Better panic reporting to C FFI
  • Backport fixes to brotli issues 502 and 506

What's new in version 2.0.0

  • Legacy Custom dictionaries (mostly useful for testing multithreaded brotli encoding and experimentation)
  • New mechanism to request that the library should only depend on custom allocations: the --no-default-features flag since --features=std is on by default.
  • Fully compatible C FFI to match the https://github.com/google/brotli C API and become a drop-in replacement

Project Requirements

Direct no-stdlib port of the C brotli decompressor to Rust if --no-default-features is passed into the build

no dependency on the Rust stdlib: this library would be ideal for decompressing within a rust kernel among other things.

This will be useful to see how C and Rust compare in an apples-to-apples comparison where the same algorithms and data structures and optimizations are employed.

The current expected performance losses come from

  1. an extra indirection in the hgroups
  2. array bounds checks on every access
  3. no ability to load a full aligned 64 bit or 128 bit item from a [u8]

the system also enables all syscalls to be "frontloaded" in the initial generation of a memory pool for the allocator. Afterwards, SECCOMP can be activated or other mechanisms can be used to secure the application, if desired

Linking rust-brotli-decompressor with C code using the zero-cost rust FFI abstraction

This library has FFI exports which comply with the original C interfaces. To build them, enter the c directory and just type make there. That will build a small example program and the cdylib with the appropriate ffi in place to link against

the example, called c/main.c shows how to decompress a program using the streaming interface and the nonstreaming interface.

If a nostdlib version is desired, then an unstable rust must be used (to enable the custom panic handler) and then the BrotliDecoderDecompress function is deactivated since that has no facilities for specifying a custom malloc

a customized malloc must be used if a nostdlib build is chosen and additionally the no-stdlib-ffi-binding cargo feature must be set eg

cargo build --features='no-stdlib no-stdlib-ffi-binding' --release

Usage

With the io::Read abstraction

let mut input = brotli_decompressor::Decompressor::new(&mut io::stdin(), 4096 /* buffer size */);

then you can simply read input as you would any other io::Read class

With the Stream Copy abstraction

match brotli_decompressor::BrotliDecompress(&mut io::stdin(), &mut io::stdout(), 65536 /* buffer size */) {
    Ok(_) => {},
    Err(e) => panic!("Error {:?}", e),
}

With manual memory management

There are 3 steps to using brotli without stdlib

  1. setup the memory manager
  2. setup the BrotliState
  3. in a loop, call BrotliDecompressStream

in Detail

// at global scope declare a MemPool type -- in this case we'll choose the heap to
// avoid unsafe code, and avoid restrictions of the stack size

declare_stack_allocator_struct!(MemPool, heap);

// at local scope, make a heap allocated buffers to hold uint8's uint32's and huffman codes
let mut u8_buffer = define_allocator_memory_pool!(4096, u8, [0; 32 * 1024 * 1024], heap);
let mut u32_buffer = define_allocator_memory_pool!(4096, u32, [0; 1024 * 1024], heap);
let mut hc_buffer = define_allocator_memory_pool!(4096, HuffmanCode, [0; 4 * 1024 * 1024], heap);
let heap_u8_allocator = HeapPrealloc::<u8>::new_allocator(4096, &mut u8_buffer, bzero);
let heap_u32_allocator = HeapPrealloc::<u32>::new_allocator(4096, &mut u32_buffer, bzero);
let heap_hc_allocator = HeapPrealloc::<HuffmanCode>::new_allocator(4096, &mut hc_buffer, bzero);

// At this point no more syscalls are going to be needed since everything can come from the allocators.

// Feel free to activate SECCOMP jailing or other mechanisms to secure your application if you wish.

// Now it's possible to setup the decompressor state
let mut brotli_state = BrotliState::new(heap_u8_allocator, heap_u32_allocator, heap_hc_allocator);

// at this point the decompressor simply needs an input and output buffer and the ability to track
// the available data left in each buffer
loop {
    result = BrotliDecompressStream(&mut available_in, &mut input_offset, &input.slice(),
                                    &mut available_out, &mut output_offset, &mut output.slice_mut(),
                                    &mut written, &mut brotli_state);

    // just end the decompression if result is BrotliResult::ResultSuccess or BrotliResult::ResultFailure
}

This interface is the same interface that the C brotli decompressor uses

Also feel free to use custom allocators that invoke Box directly. This example illustrates a mechanism to avoid subsequent syscalls after the initial allocation