Pithos library

pithos_lib is the public Rust API for creating, opening, reading, extending, extracting, and adapting Pithos archives. It has no network client or transport policy. Cargo package versioning is separate.

Installation

Add the crate to an application using Rust 1.89 or newer:

[dependencies]
pithos_lib = "0.8"

The default features ro-crate and crypt4gh enable the RO-Crate conversion and Crypt4GH export adapters. Use default-features = false to build only the archive format, reader, writer and Linux filesystem operations.

Create an archive

A base archive needs no keys and accepts only local, uncompressed, unencrypted content:

use pithos_lib::archive::{ArchivePath, ArchiveWriter, EntryMetadata, ProcessingOptions, WriteOptions};
use std::fs::File;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut writer = ArchiveWriter::create(File::create("report.pith")?, WriteOptions::base())?;
    writer.add_file(
        ArchivePath::new("report.txt")?,
        EntryMetadata::new(0, 0, 0o644),
        ProcessingOptions::new(false, 0)?,
        None,
        File::open("report.txt")?,
    )?;
    writer.finish()?;
    Ok(())
}

This example encrypts a local file for its owner. The recipient public key is derived from the private key only for a self-contained example; production applications should provide the intended recipients' public keys.

use pithos_lib::archive::{
    ArchivePath, ArchiveWriter, EntryMetadata, ProcessingOptions, WriteOptions,
};
use pithos_lib::crypto::PrivateKey;
use std::fs::File;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let owner = PrivateKey::from_private_pem_bytes(&std::fs::read("owner.pem")?)?;
    let recipient = owner.public_key();
    let input = File::open("report.txt")?;
    let size = input.metadata()?.len();
    let output = File::create("report.pith")?;

    let mut writer = ArchiveWriter::create(output, WriteOptions::new(owner, vec![recipient]))?;
    writer.add_file(
        ArchivePath::new("report.txt")?,
        EntryMetadata::new(0, 0, 0o644),
        ProcessingOptions::default(),
        Some(size),
        input,
    )?;
    writer.finish()?;
    Ok(())
}

See examples/create.rs for the maintained version and the CLI guide for a command-line workflow.

Public API

Create archives with archive::ArchiveWriter, archive::WriteOptions, and crypto keys.
Open an immutable archive::Archive from a source::ArchiveSource, such as source::FileSource or source::MemorySource.
Read a complete entry with copy_to or a checked range with copy_range_to.
Use fs::extract, fs::append_files, fs::grant_readers, and fs::ingest for Linux host operations.
Use adapters::ro_crate for local directory/ZIP conversion and adapters::crypt4gh::export for Crypt4GH output.
Seal parts of one file independently with archive::PieceEncoder and join the stored parts later with archive::compose, which opens no key. Keep Composition::metadata_digest in trusted storage and pass it to OpenOptions::with_expected_metadata_digest.

Compiled examples are included with the package: create, open_list, read_ranges, extract, append, grant_readers, ro_crate, and crypt4gh. They are local-only and return errors to their caller.

The typed archive API deliberately keeps old model, helper, and internal format access private. Use WriteOptions::base for plain output or WriteOptions::new for encrypted output.

Read and extract semantics

Opening validates archive structure and accessible metadata. Payload integrity is lazy: each block is verified when it is read. copy_to and copy_range_to do not write bytes from a failed block, but a generic sink can retain earlier verified blocks if a later block fails. PithosError::ContentUnavailable means the entry is present but no supplied key can access its content; it is different from an integrity or corruption error.

fs::extract is Linux-only. It walks destination components without following symlinks, stages regular-file output anonymously with Linux O_TMPFILE, and publishes without replacing an existing entry. The destination filesystem must support O_TMPFILE and linkat(AT_EMPTY_PATH); extraction can therefore fail on a collision rather than overwrite a destination.

Async reading

The optional async feature adds source::AsyncArchiveSource, archive::AsyncArchive and archive::AsyncExternalBlockResolver. It depends only on futures-core and needs no specific async runtime.

AsyncArchive::open(source, options, known_len) issues the header, footer and directory requests one after the other. Their count does not depend on the number of blocks. Pass the object length as known_len to skip the length request.
read_range(path, range) returns a Stream of verified plaintext chunks in file order. ReadLimits bounds the outstanding requests and the bytes held for responses, decoding and ordered delivery. Dropping the stream cancels its outstanding requests.
read_range_owned(path, range) takes the archive in an Arc and returns an OwnedRangeStream that holds it. It reads like read_range, with the same order, bounds and cancellation, but it is Send + 'static. A server handler can return it as a streaming response body, for example after mapping its error type.
The source must serve one immutable object revision for the open and every later read, and each response must have exactly the requested length.
Decoding directories and blocks is CPU work. open runs it inline, which suits only small archives. Pass a BlockingHook to open_with_hook to move it to a blocking pool, for example one that forwards to tokio::task::spawn_blocking.
External blocks follow the rules of the synchronous reader: they are read only when both a resolver and an access policy are supplied, and the resolver checks the policy before its initial access and before every redirect.

A handler that shares one opened archive can return a body stream like this:

use futures_core::Stream;
use pithos_lib::archive::AsyncArchive;
use pithos_lib::error::PithosError;
use pithos_lib::source::AsyncArchiveSource;
use std::ops::Range;
use std::sync::Arc;

fn body<S: AsyncArchiveSource + 'static>(
    archive: &Arc<AsyncArchive<S>>,
    path: &str,
    range: Range<u64>,
) -> Result<impl Stream<Item = Result<Vec<u8>, PithosError>> + Send + 'static, PithosError> {
    Arc::clone(archive).read_range_owned(path, range)
}

RO-Crate ingestion

Load a directory or ZIP RO-Crate, then convert its retained source through a configured writer:

use pithos_lib::adapters::ro_crate::{read_ro_crate_directory, write_ro_crate};
use pithos_lib::archive::ProcessingOptions;

# fn convert<W: std::io::Write>(writer: &mut pithos_lib::archive::ArchiveWriter<W>) -> Result<(), Box<dyn std::error::Error> {
let loaded = read_ro_crate_directory("path/to/ro-crate")?;
write_ro_crate(writer, loaded, ProcessingOptions::default())?;
# Ok(())
# }

The upstream parser accepts RO-Crate 1.1 and 1.2 metadata without upstream validation or warning emission; Pithos then applies its own source, path, limit, metadata, and conversion policy. Conversion stores the inspected ro-crate-metadata.json bytes rather than reserializing the graph. ZIP conversion streams retained members and does not extract them first.

Memory wiping

Pithos wipes its protected internal key and transform buffers. They hold keys, key lists, and block content while Pithos chunks, compresses, encrypts, decrypts, decompresses and verifies it. Other buffers are not wiped. For example, MemorySource keeps the archive bytes in a plain Arc<[u8]>, which holds the plaintext of plain blocks. Plaintext that a read hands to the caller is the caller's to wipe. zstd internal allocations may retain plaintext after being freed.

Platform behavior

pithos_lib builds only on Linux, even with default-features = false. The Linux filesystem modules in fs and the Unix file source in source are always compiled.

Filesystem ingestion, extraction, append, and grants are Linux-only in 0.8. Append uses a cooperating-writer advisory lock and can request AppendDurability::SyncAll; it attempts to truncate a failed child directory, but it cannot promise rollback after an unrecoverable host or power failure.

pithos_lib 0.8.0