chapter-tgz

This is a library for creating and consuming specially crafted .tar.gz files with the following properties:

1. Efficient access to specific predefined points in the tar ("chapter boundaries"). A chapter consists of zero or more consecutive tar entries. This can be used to skip over groups of tar entries in O(1) time without performing the work of gzip decompression on the intervening entries.

2. Parallel decompression: different chapters of the same tgz can be read simultaneously by different threads. Extracting a later entry is not stalled on processing all previous entries as in a conventional tgz file.

3. Perfectly compatible with existing readers that do not know about chapter information. All existing software will be able to read these files as ordinary tgz files. Chapter information is encoded in the form of valid empty gzip blocks with peculiar Huffman code alphabets.

4. Perfectly compatible with existing writers that do not embed chapter information. Tgz files without chapter information are handled as if there was a single chapter encompassing all of their entries.

Examples

Writing: this example demonstrates creating a tgz with several large files containing random data, each in its own chapter. (In general a chapter may consist of zero or more tar entries.)

use chapter_tgz::{Compression, TgzWriter};
use rand::RngReader;
use rand::rngs::SmallRng;
use std::fs;
use std::io::{self, Read as _};

fn main() -> io::Result<()> {
    let mut tgz = TgzWriter::new(Vec::new(), Compression::fast());
    let mut rng: SmallRng = rand::make_rng();
    for i in 0..20 {
        let mut chapter = tgz.create_chapter();
        let mut header = tar::Header::new_gnu();
        header.set_size(100_000_000);
        let path = format!("random/{i}");
        let data = RngReader(&mut rng).take(100_000_000);
        chapter.append_data(&mut header, path, data)?;
    }
    let compressed = tgz.into_inner()?;
    fs::write("example.tar.gz", compressed)?;
    Ok(())
}

Reading: this example demonstrates decompressing tgz chapters in parallel.

use chapter_tgz::TgzReader;
use std::fs;
use std::io::{self, Cursor};
use std::thread;

fn main() -> io::Result<()> {
    let data = fs::read("example.tar.gz")?;
    let tgz = TgzReader::open(Cursor::new(data))?;

    let n = tgz.chapters();
    let mut chapters = Vec::with_capacity(n as usize);
    for i in 0..n {
        let mut chapter = tgz.independent_read_chapter(i)?;
        if i % 2 == 0 {
            // Option 1: We can directly enqueue chapters into the thread pool.
            chapters.push(chapter);
        } else if let Some(first_entry) = chapter.entries()?.next()
            && let Some(file_name) = first_entry?.path()?.file_name()
            && let Some(file_name_str) = file_name.to_str()
            && !file_name_str.starts_with("__")
        {
            // Option 2: We can examine chapter entries to decide whether to
            // process or skip a chapter. Reading the first entry's tar header
            // (path, size, PAX extensions) from a chapter is fast.
            chapters.push(tgz.independent_read_chapter(i)?);
        }
    }

    thread::scope(|scope| {
        for mut chapter in chapters {
            scope.spawn(move || {
                if let Err(err) = (|| -> io::Result<()> {
                    for _entry in chapter.entries()? {}
                    Ok(())
                })() {
                    eprintln!("Error: {err}");
                }
            });
        }
    });

    Ok(())
}

License