Crate ztensor 

zTensor v2: an aligned, verifiable container format for tensor data.

zTensor carries what can be proved about a checkpoint’s bytes: where they are (Part::locate), whether they are intact (Part::verify), whether they can be dropped without disturbing a neighbour (Part::evict), and which model they are (Manifest::content_digest).

use ztensor::{DType, Source, Writer};

let mut w = Writer::create("model.zt")?;
w.add("weights", [2u64, 2], DType::F32, &[0u8; 16])?;
w.finish()?;

let src = Source::open("model.zt")?;
let t = src.tensor("weights")?;
let bytes = t.data()?.map()?;      // borrowed, or an error; never a copy

The modules are the spec’s layers

The format is specified in spec/ztensor-v2-spec.md (Draft 4), which separates three layers, and this crate is laid out to match so that a spec section and the code that implements it have the same address:

• format — L0 container and L1 manifest, frozen. The magic, the 40-byte footer, the alignment floor, the manifest schema and its CBOR mapping, and the rules that decide conformance and canonical form. Nothing here opens a file.
• vocab — L2, open and registry-managed: layouts, logical types and encodings, which another crate can extend and which are then validated exactly like the built-ins.
• read — opening .zt and getting at bytes.
• write — producing .zt.
• provide — the face turned towards a crate that projects a foreign format into a Source. Reading a checkpoint needs nothing from it.

The names a consumer actually uses are re-exported at the crate root, and only those: the format constants, the builder types you never have to name, the shard resolvers, the free functions that take a path instead of a source, and the projection machinery all keep their module paths.

Getting bytes

Bytes belong to a Part, never to a Tensor: a dense tensor keeps all of them in its "data" part, a quantized one spreads them over that part and its scales. Tensor::data is the step between, and it is a step rather than a shorthand so that which part is being addressed never goes unsaid.

Then three methods, one per intent: bytes gives the best the source can do as a Cow that says which it gave, map insists on a borrow, and locate gives the address so the caller can do the I/O itself. Caps reports per part what each will do, and whose fields are named after those very methods.

Two layers of description

Manifest is what one .zt file literally says. Catalog is the resolved index a consumer queries, whose addresses are StoreIds and which can therefore span files that never heard of each other. Foreign projections build catalogs; only a .zt root has a manifest, which is what Source::provenance reports.

Re-exports

pub use format::DType;

pub use format::DigestAlgorithm;

pub use format::Manifest;

pub use format::Object;

pub use format::Shard;

pub use provide::Location;

pub use provide::Store;

pub use provide::StoreId;

pub use read::Caps;

pub use read::Part;

pub use read::Provenance;

pub use read::Source;

pub use read::Tensor;

pub use read::Verified;

pub use vocab::Vocabulary;

pub use write::Sink;

pub use write::Writer;

Modules

format

The frozen layers: L0 container and L1 manifest.

provide

Building a Source out of something that is not a .zt.

read

Reading tensors.

vocab

L2 vocabulary: layouts, encodings and logical types, held as a value.

write

.zt writing.

Enums

Error

Rule

The spec rule a rejected file violated.

Type Aliases

Result