# Pithos library
`pithos_lib` is the public Rust API for creating, opening, reading, extending, extracting, and adapting Pithos archives. It has no network client or transport policy. Cargo package versioning is separate.
## Installation
Add the crate to an application using Rust 1.89 or newer:
```toml
[dependencies]
pithos_lib = "0.8"
```
The default features `ro-crate` and `crypt4gh` enable the RO-Crate conversion and Crypt4GH
export adapters. Use `default-features = false` to build only the archive format, reader,
writer and Linux filesystem operations.
## Create an archive
A base archive needs no keys and accepts only local, uncompressed, unencrypted content:
```rust,no_run
use pithos_lib::archive::{ArchivePath, ArchiveWriter, EntryMetadata, ProcessingOptions, WriteOptions};
use std::fs::File;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut writer = ArchiveWriter::create(File::create("report.pith")?, WriteOptions::base())?;
writer.add_file(
ArchivePath::new("report.txt")?,
EntryMetadata::new(0, 0, 0o644),
ProcessingOptions::new(false, 0)?,
None,
File::open("report.txt")?,
)?;
writer.finish()?;
Ok(())
}
```
This example encrypts a local file for its owner. The recipient public key is derived from the private key only for a self-contained example; production applications should provide the intended recipients' public keys.
```rust,no_run
use pithos_lib::archive::{
ArchivePath, ArchiveWriter, EntryMetadata, ProcessingOptions, WriteOptions,
};
use pithos_lib::crypto::PrivateKey;
use std::fs::File;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let owner = PrivateKey::from_private_pem_bytes(&std::fs::read("owner.pem")?)?;
let recipient = owner.public_key();
let input = File::open("report.txt")?;
let size = input.metadata()?.len();
let output = File::create("report.pith")?;
let mut writer = ArchiveWriter::create(output, WriteOptions::new(owner, vec![recipient]))?;
writer.add_file(
ArchivePath::new("report.txt")?,
EntryMetadata::new(0, 0, 0o644),
ProcessingOptions::default(),
Some(size),
input,
)?;
writer.finish()?;
Ok(())
}
```
See [`examples/create.rs`](examples/create.rs) for the maintained version and the [CLI guide](../pithos/README.md) for a command-line workflow.
## Public API
- Create archives with `archive::ArchiveWriter`, `archive::WriteOptions`, and `crypto` keys.
- Open an immutable `archive::Archive` from a `source::ArchiveSource`, such as `source::FileSource` or `source::MemorySource`.
- Read a complete entry with `copy_to` or a checked range with `copy_range_to`.
- Use `fs::extract`, `fs::append_files`, `fs::grant_readers`, and `fs::ingest` for Linux host operations.
- Use `adapters::ro_crate` for local directory/ZIP conversion and `adapters::crypt4gh::export` for Crypt4GH output.
- Seal parts of one file independently with `archive::PieceEncoder` and join the stored parts later with `archive::compose`, which opens no key. Keep `Composition::metadata_digest` in trusted storage and pass it to `OpenOptions::with_expected_metadata_digest`.
Compiled examples are included with the package: [`create`](examples/create.rs), [`open_list`](examples/open_list.rs), [`read_ranges`](examples/read_ranges.rs), [`extract`](examples/extract.rs), [`append`](examples/append.rs), [`grant_readers`](examples/grant_readers.rs), [`ro_crate`](examples/ro_crate.rs), and [`crypt4gh`](examples/crypt4gh.rs). They are local-only and return errors to their caller.
The typed archive API deliberately keeps old model, helper, and internal format access private. Use `WriteOptions::base` for plain output or `WriteOptions::new` for encrypted output.
## Read and extract semantics
Opening validates archive structure and accessible metadata. Payload integrity is lazy: each block is verified when it is read. `copy_to` and `copy_range_to` do not write bytes from a failed block, but a generic sink can retain earlier verified blocks if a later block fails. `PithosError::ContentUnavailable` means the entry is present but no supplied key can access its content; it is different from an integrity or corruption error.
`fs::extract` is Linux-only. It walks destination components without following symlinks, stages regular-file output anonymously with Linux `O_TMPFILE`, and publishes without replacing an existing entry. The destination filesystem must support `O_TMPFILE` and `linkat(AT_EMPTY_PATH)`; extraction can therefore fail on a collision rather than overwrite a destination.
## Async reading
The optional `async` feature adds `source::AsyncArchiveSource`, `archive::AsyncArchive` and
`archive::AsyncExternalBlockResolver`. It depends only on `futures-core` and needs no specific
async runtime.
- `AsyncArchive::open(source, options, known_len)` issues the header, footer and directory
requests one after the other. Their count does not depend on the number of blocks. Pass the
object length as `known_len` to skip the length request.
- `read_range(path, range)` returns a `Stream` of verified plaintext chunks in file order.
`ReadLimits` bounds the outstanding requests and the bytes held for responses, decoding and
ordered delivery. Dropping the stream cancels its outstanding requests.
- `read_range_owned(path, range)` takes the archive in an `Arc` and returns an
`OwnedRangeStream` that holds it. It reads like `read_range`, with the same order, bounds and
cancellation, but it is `Send + 'static`. A server handler can return it as a streaming
response body, for example after mapping its error type.
- The source must serve one immutable object revision for the open and every later read, and
each response must have exactly the requested length.
- Decoding directories and blocks is CPU work. `open` runs it inline, which suits only small
archives. Pass a `BlockingHook` to `open_with_hook` to move it to a blocking pool, for example
one that forwards to `tokio::task::spawn_blocking`.
- External blocks follow the rules of the synchronous reader: they are read only when both a
resolver and an access policy are supplied, and the resolver checks the policy before its
initial access and before every redirect.
A handler that shares one opened archive can return a body stream like this:
```rust,no_run
use futures_core::Stream;
use pithos_lib::archive::AsyncArchive;
use pithos_lib::error::PithosError;
use pithos_lib::source::AsyncArchiveSource;
use std::ops::Range;
use std::sync::Arc;
fn body<S: AsyncArchiveSource + 'static>(
archive: &Arc<AsyncArchive<S>>,
path: &str,
range: Range<u64>,
) -> Result<impl Stream<Item = Result<Vec<u8>, PithosError>> + Send + 'static, PithosError> {
Arc::clone(archive).read_range_owned(path, range)
}
```
## RO-Crate ingestion
Load a directory or ZIP RO-Crate, then convert its retained source through a configured writer:
```rust,no_run
use pithos_lib::adapters::ro_crate::{read_ro_crate_directory, write_ro_crate};
use pithos_lib::archive::ProcessingOptions;
# fn convert<W: std::io::Write>(writer: &mut pithos_lib::archive::ArchiveWriter<W>) -> Result<(), Box<dyn std::error::Error> {
let loaded = read_ro_crate_directory("path/to/ro-crate")?;
write_ro_crate(writer, loaded, ProcessingOptions::default())?;
# Ok(())
# }
```
The upstream parser accepts RO-Crate 1.1 and 1.2 metadata without upstream validation or warning emission; Pithos then applies its own source, path, limit, metadata, and conversion policy. Conversion stores the inspected `ro-crate-metadata.json` bytes rather than reserializing the graph. ZIP conversion streams retained members and does not extract them first.
## Memory wiping
Pithos wipes its protected internal key and transform buffers. They hold keys, key lists, and block content while Pithos chunks, compresses, encrypts, decrypts, decompresses and verifies it. Other buffers are not wiped. For example, `MemorySource` keeps the archive bytes in a plain `Arc<[u8]>`, which holds the plaintext of plain blocks. Plaintext that a read hands to the caller is the caller's to wipe. zstd internal allocations may retain plaintext after being freed.
## Platform behavior
`pithos_lib` builds only on Linux, even with `default-features = false`. The Linux filesystem modules in `fs` and the Unix file source in `source` are always compiled.
Filesystem ingestion, extraction, append, and grants are Linux-only in 0.8. Append uses a cooperating-writer advisory lock and can request `AppendDurability::SyncAll`; it attempts to truncate a failed child directory, but it cannot promise rollback after an unrecoverable host or power failure.