Crate composefs_splitdirfdstream

Expand description

A data format and IPC protocol for sending a binary stream across local processes via file descriptor passing (DBus, varlink, etc.).

Designed for sending tar archives of container image layers that are unpacked into a storage system such as composefs or docker/podman overlay storage. More generally it is a mechanism for reassembling any byte stream from a mix of inline bytes and whole-file references, useful whenever the bulk of a stream already lives as files in a content store and copying that bulk inline would be wasteful.

External content is identified by (dirfd_index, filename) pairs so the receiving side can openat2 the files itself given a small out-of-band array of directory file descriptors. This avoids passing one open fd per external chunk (which would not scale to streams referencing thousands of files) and suits reconstructing a stream from a content store laid out on disk.

§Format

A splitdirfdstream is a sequence of chunks with no header or footer. Combined with an out-of-band, ordered array of directory file descriptors dirfds[0..D], it reconstructs a byte stream by concatenating each chunk’s contribution:

Metadata chunk — raw stream metadata (tar header/padding) carried verbatim.
InlineData chunk — file content transported inline (for non-world-readable files the producer read through a privileged fd).
FileBackedData chunk — content_length bytes of a file, resolved via openat2(dirfds[dirfd_index], filename, RESOLVE_BENEATH) and read from offset 0.

All integers are little-endian. Each chunk begins with a single type byte:

Type byte	Chunk	Remaining header	Body
`0x00`	Metadata	`u32 LE` — body length	`length` bytes
`0x01`	InlineData	`u32 LE` — body length	`length` bytes
`0x02`	FileBackedData	`u64 LE` content_length, `u32 LE` dirfd_index, `u32 LE` name_len	`name_len` bytes of filename

Any other type byte is a hard error (Error::UnknownChunkType).

There is no in-band end-of-stream sentinel: the stream ends at clean EOF at the start of a type byte. A partial read anywhere inside a chunk is a truncation error (Error::Truncated). Because the format carries no length or checksum of the whole stream, callers that need end-to-end integrity should verify the reconstructed bytes against an expected size and digest out of band.

§Chunk layouts

§Metadata

+--------+---------------+----------------------------+
| 0x00   | length: u32LE | data: `length` raw bytes   |
+--------+---------------+----------------------------+

data (length bytes) is written verbatim to output (tar headers, padding, etc.). Empty writes are silently dropped by the writer; a zero-length Metadata chunk is never encoded. length is bounded by MAX_INLINE_CHUNK_SIZE (256 MiB).

§InlineData

+--------+---------------+----------------------------+
| 0x01   | length: u32LE | data: `length` raw bytes   |
+--------+---------------+----------------------------+

length is both the byte count of the data that follows and the logical file size the consumer uses for its inline-vs-object storage decision. Unlike FileBackedData, no directory fd is involved; the producer has already read the content through its own privileged fd.

A zero-length InlineData is written and round-trips correctly — a zero-byte non-world-readable file must still be transported. length is bounded by MAX_INLINE_CHUNK_SIZE (256 MiB), since the data is fully buffered in memory during transport.

§FileBackedData

+--------+---------------------+--------------------+-----------------+------------------------------+
| 0x02   | content_len: u64LE  | dirfd_index: u32LE | name_len: u32LE | filename: name_len raw bytes |
+--------+---------------------+--------------------+-----------------+------------------------------+

The consumer reads exactly content_len bytes starting at offset 0 of the opened file. The explicit content_len makes the stream self-framing — entry boundaries never depend on trusting the backing file’s size, which closes a TOCTOU window if the underlying store is mutated mid-read.

A FileBackedData chunk always starts at offset 0: it references a whole file, not a byte range. This is deliberate — it lets every external reference reflink cleanly (FICLONE) when materialized on a CoW filesystem. There is no range variant.

filename is a path relative to dirfds[dirfd_index]. It may contain / to traverse subdirectories within that root; it is not NUL-terminated and must not contain NUL.

§Limits

Constant	Value	Purpose
`MAX_INLINE_CHUNK_SIZE`	256 MiB	Bounds memory for a single Metadata or InlineData chunk body.
`MAX_FILENAME_LEN`	4096 bytes	Bounds the FileBackedData filename length.

The reader rejects an out-of-range dirfd_index, and Metadata or InlineData chunks whose length exceeds MAX_INLINE_CHUNK_SIZE.

§Safety

The consuming side should always use openat2(RESOLVE_BENEATH) or equivalent. This crate uses rustix::fs::openat2 with RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS | RESOLVE_NO_MAGICLINKS, falling back to openat(O_NOFOLLOW) on kernels older than 5.6. The validate_filename function rejects .. components, absolute paths, and embedded NUL bytes before any syscall is made.

The reader performs only structural validation (framing, limits); it does not validate filename content. Callers that consume Chunk::FileBackedData .filename directly must call validate_filename (or use open_beneath / reconstruct, which do).

§Examples

Write a stream, then inspect its chunk structure:

use composefs_splitdirfdstream::{SplitdirfdstreamWriter, SplitdirfdstreamReader, Chunk};

// Write a stream: inline glue bytes plus a reference to dirfds[0]/data/blob.
let mut buffer = Vec::new();
let mut writer = SplitdirfdstreamWriter::new(&mut buffer);
writer.write_metadata(b"tar header bytes").unwrap();
writer.write_file_backed_data(0, 5, b"data/blob").unwrap(); // 5 bytes from dirfds[0]/data/blob
writer.write_metadata(b"tar padding").unwrap();
writer.finish().unwrap();

// Inspect the chunk structure.
let mut reader = SplitdirfdstreamReader::new(buffer.as_slice());
while let Some(chunk) = reader.next_chunk().unwrap() {
    match chunk {
        Chunk::Metadata(data) => { /* tar header/padding bytes */ }
        Chunk::InlineData(data) => { /* inline file content (non-world-readable files) */ }
        Chunk::FileBackedData { dirfd_index, length, filename } => { /* (dirfds[i], name, len) */ }
    }
}

To reconstruct the full byte stream, supply the directory fds to reconstruct, which resolves and splices each external chunk for you:

use std::os::fd::BorrowedFd;
let dirfds = [dir];
let total = composefs_splitdirfdstream::reconstruct(stream, &dirfds, out).unwrap();

§API surface

Item	Role
`SplitdirfdstreamWriter`	Encode inline + external chunks into the wire format.
`SplitdirfdstreamReader`	Iterate a stream as borrowed `Chunk`s.
`Chunk`	`Metadata(&[u8])`, `InlineData(&[u8])`, or `FileBackedData { dirfd_index, length, filename }`.
`reconstruct`	Reconstruct the full byte stream given the directory fds.
`open_beneath`	Safely open one external file beneath a directory fd.
`validate_filename`	The path-safety predicate (reused by the writer/consumer).

§See also

This crate is only the stream format. A higher-level control channel — opening a source, negotiating capabilities, and handing over the stream fd plus directory fds over a socket via SCM_RIGHTS — is layered on top elsewhere (e.g. the composefs-storage layer-transfer service); it carries structured metadata only, with all binary content flowing through this format and the directory fds.

Re-exports§

pub use transport::FdLimitError;
pub use transport::LayerFdLayout;
pub use transport::MAX_FDS_PER_FRAME;
pub use transport::build_layer_fd_layout;
pub use transport::open_devnull;
pub use transport::seed_from_id;
pub use transport::split_fds_into_frames;

Modules§

transport: Source-agnostic FD-transport mechanics for the splitdirfdstream wire protocol.

Structs§

SplitdirfdstreamReader: Reader for parsing a splitdirfdstream.
SplitdirfdstreamWriter: Writer for building a splitdirfdstream.

Enums§

Chunk: A chunk decoded from a splitdirfdstream.
Error: Errors that can occur while reading or writing a splitdirfdstream.

Constants§

MAX_FILENAME_LEN: Maximum length of an external filename in bytes.
MAX_INLINE_CHUNK_SIZE: Maximum size for an inline chunk (256 MiB).

Functions§

open_beneath: Open a file at filename relative to dirfd using safe kernel primitives.
reconstruct: Reconstruct the byte stream encoded in stream by combining inline chunks and external file data, writing all output to output.
validate_filename: Validate that filename is an acceptable external filename.

Type Aliases§

Result: Convenience alias for Result<T, Error>.