Crate tsaur_core 

tsaur-core

Reference implementation of the T-saur archive format (.tsr): a content-addressed, agent-first archive.

• content-defined chunking (FastCDC) + BLAKE3 identities + deduplication, Merkle root identity
• block mode (default), solid blocks or granular blobs; codecs zstd, zstd + trained dictionary, xz (LZMA2) and PPMd H, chosen per block; x86 and ARM64 branch filters for machine code
• container awareness: ZIP/OPC members and PDF FlateDecode streams inverted with preflate, JPEG (files and DCTDecode streams) recoded losslessly with Lepton; always rebuilt bit-exact
• compression by reference: chunks present in a referenced archive are not stored again
• per-blob authenticated encryption (XChaCha20-Poly1305) with an envelope key unlocked by Argon2id passwords or hybrid X25519 + ML-KEM-768 recipients; Ed25519 signatures
• hard resource limits (anti decompression bomb), path sanitisation by construction
• streaming packer and reader (bounded memory), range reads, canonical views (DOCX, PDF)
• fixed-size pieces + Reed-Solomon parity sidecars for transport verification and recovery

The format is specified in docs/spec/; tsaur (the CLI crate) adds the agent-facing commands and the MCP server.

Re-exports

pub use error::Error;

pub use error::Result;

pub use pack::pack;

pub use pack::PackOptions;

pub use pack::PackReport;

pub use read::inspect;

pub use read::Inspect;

pub use read::Reader;

pub use read::VerifyReport;

Modules

canonical

Canonical (semantic) views for agents: the information of a document as Markdown/text, explicitly not bit-exact (spec §7, §11). DOCX is converted from its own XML, PDF through text extraction. Every view starts with a short provenance header so an agent knows what it is looking at.

chunk

Content-defined chunking (FastCDC 2020, normalized chunking level 2) and BLAKE3 identities.

codec

Compression codecs. v0.1 ships store, zstd, zstd-with-dictionary and xz (LZMA2); the codec id space is reserved for context mixing, model-predictive, delta and float-split codecs (spec §4.2).

container

Container awareness for ZIP-based formats (DOCX/XLSX/PPTX/EPUB/JAR/ODT/plain ZIP).

crypto

Envelope encryption and signatures.

error

Error type and stable exit codes.

format

On-disk framing: fixed header, section table, trailer, canonical CBOR helpers.

manifest

Manifest and chunk index data model (serialised as CBOR; field order is fixed by the structs, which keeps the encoding deterministic for identical inputs).

pack

Packing: enumerate → (invert containers) → chunk → dedup → group into blocks → compress → seal → stream to disk. Memory stays bounded by the block size and the chunk index: file contents are chunked with a streaming FastCDC reader, unique chunks are compressed in blocks and written as soon as a batch is ready, and only container files (which need preflate) are held in memory.

paths

Archive paths are labels, never destinations. They are normalised on the way in and validated again on the way out, so an archive can never write outside the target directory.

pieces

Transport pieces and recovery: the archive file is cut into fixed pieces; a .pieces sidecar (like a .torrent) lists BLAKE3 hashes and a Merkle root; a .par sidecar holds systematic Reed-Solomon parity (GF(2^8), stripes of at most 200 data pieces). Any k of the k + m pieces of a stripe rebuild the stripe. Parity symbols are hashed too: FEC alone never authenticates data.

read

Reading: open → verify framing and every section hash → (unwrap key) → decode index and manifest → serve chunks, ranges, entries; verify; extract with safe paths.

tls

Encrypted, peer-authenticated connections for the volume exchange: TLS 1.3 (rustls with the ring provider) between two T-saur instances that each hold a self-signed certificate. There is no certificate authority, account or server of ours: identities are pinned locally by the SHA-256 fingerprint of the certificate, which the user copies through a channel they trust (the same channel that carries the set id). A connection is accepted only when the remote certificate hashes to a pinned fingerprint; everything else in the certificate (names, dates, issuer) is ignored on purpose.

transfer

Direct piece exchange between two T-saur instances (prototype): a tiny request/response protocol over TCP with manually supplied peer addresses. No discovery, no NAT traversal, no tracker, no account. The server only serves bytes of the volumes it was started with; the client verifies every descriptor and every piece against what it expects (docs/design/VOLUME-TRUST.md) and writes volumes with the offline format, so join, inspect and repair are unchanged.

volumes

Offline volume sets: one archive file striped across N data volumes plus M parity volumes (systematic Reed-Solomon over GF(2^8), one piece per volume per stripe), so that the archive can be spread over directories or removable drives and rebuilt from any N of the N + M volumes. Every volume carries the complete descriptor (geometry, archive hash, every piece hash), so no separate index file is needed. See docs/design/VOLUME-SETS.md.

Constants

VERSION

Library version string (matches the crate version).

WRITER

Identifies the writer line recorded inside archives, views and volume descriptors. It names the format (v1) and the writer generation, not the crate release, so that the same inputs give the same bytes across patch releases (the golden archives in tests/golden/ pin this).