Expand description
Xet xorb and shard adapter for the Shardline server ecosystem.
This crate provides validated xorb parsing, shard metadata normalization, reconstruction response building, and xorb transfer URL construction for the Shardline server.
§Quick start
The validation helpers are pure — they check hashes and transfer namespaces before anything touches storage:
use shardline_xet_adapter::{
build_xorb_transfer_url, validate_hash_path, validate_xorb_transfer_namespace,
};
let hash = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
// Hash paths must be 64 lowercase hex characters.
assert!(validate_hash_path(hash).is_ok());
assert!(validate_hash_path("not-a-hash").is_err());
// Transfer URLs use the `default` namespace.
assert!(validate_xorb_transfer_namespace("default").is_ok());
assert!(validate_xorb_transfer_namespace("other").is_err());
let url = build_xorb_transfer_url("http://localhost:8080/", hash);
assert_eq!(url, format!("http://localhost:8080/transfer/xorb/default/{hash}"));Parsing and validating actual xorb containers starts with
validate_serialized_xorb and decode_serialized_xorb_chunks;
reconstruction responses are built with
build_batch_reconstruction_response.
Structs§
- Authenticated
Xorb Footer - An authenticated list of member hashes and lengths. Payloads remain unverified. This deliberately cannot be passed to full-container trusted visitors.
- Batch
Reconstruction Response - Batch reconstruction response for multiple file identifiers.
- Decoded
Xorb Chunk - Decoded chunk payload paired with its validated descriptor.
- File
Reconstruction Response - File reconstruction response.
- File
Reconstruction V2Response - V2 file reconstruction response.
- Incremental
Xorb Memory - Request-owned incremental native verification. Call only under aggregate memory and CPU admission; this type bounds serialized bytes, not concurrency.
- Prepared
Shard Upload - Fully verified shard metadata which has not published any object or record. This owned value can cross a blocking-worker boundary; dropping it publishes nothing. Publication must remain with the request’s current owner.
- Prepared
Xorb File - An owned, fully verified native container ready for immutable persistence. The validation result cannot be constructed or changed by callers. Storage rechecks the serialized digest before publication. This is not a cache of the health of an already published object.
- Prepared
Xorb Memory - Owned canonical bytes and their complete validation result. Callers cannot replace the bytes or construct a validation result independently of them.
- Reconstruction
Chunk Range - End-exclusive chunk index range within one xorb.
- Reconstruction
Fetch Info - Fetch information for downloading one xorb range.
- Reconstruction
Multi Range Fetch - V2 fetch information for downloading one xorb via one or more byte ranges.
- Reconstruction
Range Descriptor - V2 byte-range descriptor for one xorb fetch entry.
- Reconstruction
Term - Ordered reconstruction term returned by the Xet reconstruction API.
- Reconstruction
UrlRange - Inclusive byte range for a ranged fetch URL.
- Shard
Upload Response - Upload result for a single shard.
- Validated
Xorb - Validated top-level metadata for a serialized xorb.
- Validated
Xorb Chunk - Validated metadata for one chunk inside a serialized xorb.
- Xorb
Upload Response - Upload result for a single xorb.
Enums§
- XetAdapter
Error - Xet adapter failure.
- Xorb
Parse Error - Failure while parsing or validating a serialized xorb.
- Xorb
Visit Error - Failure while visiting decoded xorb chunks.
Constants§
- XET_
METADATA_ ACCEPT - Accept media type used by native Xet metadata clients to disambiguate metadata routes from overlapping Hub API paths when both frontends run.
- XET_
PATH_ ROUTE - XET_
READ_ TOKEN_ ROUTE - XET_
REVISIONS_ ROUTE - XET_
REVISION_ ROUTE - XET_
TREE_ ROUTE - XET_
WRITE_ TOKEN_ ROUTE - XORB_
TRANSFER_ ROUTE
Functions§
- build_
batch_ reconstruction_ response - build_
reconstruction_ response - Errors
- build_
reconstruction_ response_ with_ metrics - Errors
- build_
xorb_ transfer_ url - Builds the xorb transfer URL for a hash on the given server base URL.
- canonical_
uploaded_ xorb_ file_ length - Returns a staged file’s stored length without buffering canonical containers. Footerless inputs use bounded file-backed normalization.
- canonical_
uploaded_ xorb_ length - Returns the stored length after the same normalization used by xorb uploads.
- decode_
authenticated_ xorb_ chunk - Decodes and authenticates exactly one selected member before returning any bytes.
- decode_
serialized_ xorb_ chunks - Decodes the packed chunk stream of a previously validated serialized xorb.
- dedup_
xorb_ candidates - Parses bounded discovery hints without reading any referenced Xorb. Callers may filter these claims against authoritative repository terms before validating source content. Claims alone never establish member integrity.
- dedupe_
shard_ mapping - Errors
- map_
xorb_ visit_ error - normalize_
owned_ uploaded_ xorb_ file - Authenticate privately owned upload staging and append any missing footer. Ownership is consumed so callers cannot accidentally mutate borrowed input. On failure the staging file is reclaimed; canonical input remains unchanged.
- normalize_
owned_ uploaded_ xorb_ file_ with_ checkpoint - Normalize owned staging with cooperative cancellation between reads. Admission and staging ownership must remain with the worker until it stops.
- normalize_
serialized_ xorb - Errors
- normalize_
uploaded_ xorb_ file - Returns a verified normalized temporary file only when the source needs it. Canonical input is read in place. The source is never modified.
- parse_
uploaded_ shard - Errors
- parse_
uploaded_ shard_ file - Parses a shard from a seekable file without copying the uploaded bytes into the process heap. Only the bounded metadata sections and normalized shard representation are retained.
- parse_
uploaded_ shard_ with_ metrics - Errors
- prepare_
owned_ xorb_ file_ with_ checkpoint - Authenticate and normalize owned staging once, preserving its validation result for persistence. Call this synchronous operation under bounded blocking admission and retain ownership until its checkpoint reports stop.
- prepare_
owned_ xorb_ memory_ with_ checkpoint - Prepare an owned native request without staging files or cloning canonical input. The caller must retain memory/CPU admission until this worker stops.
- prepare_
uploaded_ shard - Performs bounded, read-only verification of a native shard and its sources.
- prepare_
uploaded_ shard_ with_ checkpoint - Read-only preparation with a request-lifetime check between verification units. The callback must return an interrupted I/O error after request cancellation. S3 materialization checks between response frames; an already-waiting read retains the storage backend’s timeout. Local reads retain the source inode.
- prepare_
uploaded_ shard_ with_ digest_ worker - Overlap SHA-256 with source authentication using an independently admitted CPU worker. The caller must supply and retain its admission token; this function releases it only after all digest workers have stopped.
- prepare_
xorb_ memory - Fully validate and normalize a bounded in-memory upload before deriving its identity. Retain this owned result through publication instead of repeating normalization and native-content validation afterwards.
- read_
verified_ xorb_ chunk - Reads a decoded member without relying on a standalone chunk copy or trusting
derived descriptor offsets. The complete source is verified before any member
is returned. Missing containers or members return
None; corruption is an error. - reconstruction_
v2_ from_ v1 - recover_
dedup_ xorb_ descriptor - Recovers a bounded descriptor for an old chunk-to-shard mapping.
- recover_
dedup_ xorb_ descriptor_ from_ candidates - Fully validates a source selected from previously bounded discovery hints. Missing sources are skipped; source/member validation remains mandatory.
- recover_
xorb_ descriptor - Rebuilds a native descriptor by authenticating its complete immutable Xorb. The result is derived metadata, not proof of repository authorization.
- register_
uploaded_ shard_ bytes - Errors
- register_
uploaded_ shard_ bytes_ with_ member_ references - Registers a shard and supplies physical member references obtained during the existing xorb validation. No second object read or cache lookup is used. The caller must check these references atomically with metadata publication.
- register_
uploaded_ shard_ bytes_ with_ verified_ descriptors - Registers bounded shard metadata with descriptors from the same complete Xorb verification pass. The callback must publish the metadata atomically.
- register_
uploaded_ shard_ file - Registers a shard parsed directly from a temporary file. Metadata sections
are bounded by
shard_metadata_limits; the uploaded byte stream itself is never copied into a process-sized buffer. - register_
uploaded_ shard_ file_ with_ member_ references - File-backed registration with references from the same validation pass. The uploaded body is not copied into a process-sized buffer. The caller must check these references atomically with metadata publication.
- register_
uploaded_ shard_ file_ with_ verified_ descriptors - File-backed registration with verified descriptors and protected references.
- resolve_
dedupe_ shard_ object - Errors
- retained_
shard_ chunk_ hashes - Errors
- retained_
shard_ chunk_ hashes_ from_ reader - Reads retained chunk metadata from a seekable shard without requiring the caller to first materialize the entire shard in memory.
- shard_
hash_ from_ object_ key_ if_ present - Errors
- shard_
object_ key - Errors
- store_
owned_ xorb_ file - Persists an owned staged container, retaining application admission through any synchronous write that outlives cancellation of its caller.
- store_
prepared_ xorb_ file - Persist a prepared owned container without repeating its complete format validation or storing decoded copies. Immutable publication rechecks the serialized digest. Started writes keep staging and admission after caller cancellation. Independent chunk readers require published member discovery metadata and support for authenticated Xorb-backed reads.
- store_
prepared_ xorb_ memory - Publish owned prepared bytes while retaining admission through synchronous storage work. Storage checks the bound immutable commitment before publication; conflicting encodings still require complete verification of retained bytes.
- store_
uploaded_ xorb - Errors
- store_
uploaded_ xorb_ bytes - Errors
- store_
uploaded_ xorb_ file - Stores only the canonical serialized Xorb from a seekable file. Decoded members are authenticated during validation, without a second stored copy. Footerless inputs use verified file-backed normalization with bounded memory.
- store_
uploaded_ xorb_ file_ path - Errors
- store_
uploaded_ xorb_ with_ metrics - Errors
- store_
xorb_ bytes_ with_ resources - Stores verified in-memory input while retaining admission through immutable persistence and any full verification of an equivalent retained encoding.
- try_
for_ each_ serialized_ xorb_ chunk - Decodes the packed chunk stream of a previously validated serialized xorb and
passes each decoded chunk to
visitorbefore decoding the next chunk. - try_
for_ each_ serialized_ xorb_ chunk_ async - Decodes the packed chunk stream of a previously validated serialized xorb and
passes each decoded chunk to an async
visitorbefore decoding the next chunk. - try_
for_ each_ serialized_ xorb_ chunk_ async_ trusted - Async counterpart to
try_for_each_serialized_xorb_chunk_trusted. - try_
for_ each_ serialized_ xorb_ chunk_ range - Decodes and rehashes a nonempty member range of a fully validated Xorb.
- try_
for_ each_ serialized_ xorb_ chunk_ trusted - Decodes a serialized xorb after the exact immutable bytes have already been
fully validated by
validate_serialized_xorb. - validate_
hash_ path - Validates that a hash path is a 64-character lowercase hex digest.
- validate_
optional_ content_ hash - Validates an optional content hash, passing when the value is
None. - validate_
serialized_ xorb - Validates a serialized Xet xorb against the requested content hash and extracts trusted chunk metadata.
- validate_
serialized_ xorb_ footer - Authenticates a footer against an independently trusted Xorb root.
Legacy boundaries return
Noneand require complete-container validation. - validate_
serialized_ xorb_ with_ data - Fully validates a container and retains its decoded payload only when it
fits within
max_decoded_bytes. Invalid containers never return payloads. The supplied buffer is cleared and reused for the result allocation. - validate_
xorb_ transfer_ namespace - Validates the transfer namespace prefix for xorb transfer routes.
- visit_
stored_ xorb_ chunk_ hashes - Errors
- visit_
stored_ xorb_ chunk_ hashes_ if_ present - Visits members only after fully validating their stored container.
- xorb_
chunks_ cache_ hash_ from_ key_ if_ present - Extracts a Xorb hash from a legacy member-cache sidecar key.
Returns
Nonewhen the key does not match the cache-namespace pattern. - xorb_
hash_ from_ object_ key_ if_ present - Errors
- xorb_
object_ key - Errors