Skip to main content

Crate shardline_xet_adapter

Crate shardline_xet_adapter 

Source
Expand description

Xet xorb and shard adapter for the Shardline server ecosystem.

This crate provides validated xorb parsing, shard metadata normalization, reconstruction response building, and xorb transfer URL construction for the Shardline server.

§Quick start

The validation helpers are pure — they check hashes and transfer namespaces before anything touches storage:

use shardline_xet_adapter::{
    build_xorb_transfer_url, validate_hash_path, validate_xorb_transfer_namespace,
};

let hash = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";

// Hash paths must be 64 lowercase hex characters.
assert!(validate_hash_path(hash).is_ok());
assert!(validate_hash_path("not-a-hash").is_err());

// Transfer URLs use the `default` namespace.
assert!(validate_xorb_transfer_namespace("default").is_ok());
assert!(validate_xorb_transfer_namespace("other").is_err());

let url = build_xorb_transfer_url("http://localhost:8080/", hash);
assert_eq!(url, format!("http://localhost:8080/transfer/xorb/default/{hash}"));

Parsing and validating actual xorb containers starts with validate_serialized_xorb and decode_serialized_xorb_chunks; reconstruction responses are built with build_batch_reconstruction_response.

Structs§

AuthenticatedXorbFooter
An authenticated list of member hashes and lengths. Payloads remain unverified. This deliberately cannot be passed to full-container trusted visitors.
BatchReconstructionResponse
Batch reconstruction response for multiple file identifiers.
DecodedXorbChunk
Decoded chunk payload paired with its validated descriptor.
FileReconstructionResponse
File reconstruction response.
FileReconstructionV2Response
V2 file reconstruction response.
IncrementalXorbMemory
Request-owned incremental native verification. Call only under aggregate memory and CPU admission; this type bounds serialized bytes, not concurrency.
PreparedShardUpload
Fully verified shard metadata which has not published any object or record. This owned value can cross a blocking-worker boundary; dropping it publishes nothing. Publication must remain with the request’s current owner.
PreparedXorbFile
An owned, fully verified native container ready for immutable persistence. The validation result cannot be constructed or changed by callers. Storage rechecks the serialized digest before publication. This is not a cache of the health of an already published object.
PreparedXorbMemory
Owned canonical bytes and their complete validation result. Callers cannot replace the bytes or construct a validation result independently of them.
ReconstructionChunkRange
End-exclusive chunk index range within one xorb.
ReconstructionFetchInfo
Fetch information for downloading one xorb range.
ReconstructionMultiRangeFetch
V2 fetch information for downloading one xorb via one or more byte ranges.
ReconstructionRangeDescriptor
V2 byte-range descriptor for one xorb fetch entry.
ReconstructionTerm
Ordered reconstruction term returned by the Xet reconstruction API.
ReconstructionUrlRange
Inclusive byte range for a ranged fetch URL.
ShardUploadResponse
Upload result for a single shard.
ValidatedXorb
Validated top-level metadata for a serialized xorb.
ValidatedXorbChunk
Validated metadata for one chunk inside a serialized xorb.
XorbUploadResponse
Upload result for a single xorb.

Enums§

XetAdapterError
Xet adapter failure.
XorbParseError
Failure while parsing or validating a serialized xorb.
XorbVisitError
Failure while visiting decoded xorb chunks.

Constants§

XET_METADATA_ACCEPT
Accept media type used by native Xet metadata clients to disambiguate metadata routes from overlapping Hub API paths when both frontends run.
XET_PATH_ROUTE
XET_READ_TOKEN_ROUTE
XET_REVISIONS_ROUTE
XET_REVISION_ROUTE
XET_TREE_ROUTE
XET_WRITE_TOKEN_ROUTE
XORB_TRANSFER_ROUTE

Functions§

build_batch_reconstruction_response
build_reconstruction_response
Errors
build_reconstruction_response_with_metrics
Errors
build_xorb_transfer_url
Builds the xorb transfer URL for a hash on the given server base URL.
canonical_uploaded_xorb_file_length
Returns a staged file’s stored length without buffering canonical containers. Footerless inputs use bounded file-backed normalization.
canonical_uploaded_xorb_length
Returns the stored length after the same normalization used by xorb uploads.
decode_authenticated_xorb_chunk
Decodes and authenticates exactly one selected member before returning any bytes.
decode_serialized_xorb_chunks
Decodes the packed chunk stream of a previously validated serialized xorb.
dedup_xorb_candidates
Parses bounded discovery hints without reading any referenced Xorb. Callers may filter these claims against authoritative repository terms before validating source content. Claims alone never establish member integrity.
dedupe_shard_mapping
Errors
map_xorb_visit_error
normalize_owned_uploaded_xorb_file
Authenticate privately owned upload staging and append any missing footer. Ownership is consumed so callers cannot accidentally mutate borrowed input. On failure the staging file is reclaimed; canonical input remains unchanged.
normalize_owned_uploaded_xorb_file_with_checkpoint
Normalize owned staging with cooperative cancellation between reads. Admission and staging ownership must remain with the worker until it stops.
normalize_serialized_xorb
Errors
normalize_uploaded_xorb_file
Returns a verified normalized temporary file only when the source needs it. Canonical input is read in place. The source is never modified.
parse_uploaded_shard
Errors
parse_uploaded_shard_file
Parses a shard from a seekable file without copying the uploaded bytes into the process heap. Only the bounded metadata sections and normalized shard representation are retained.
parse_uploaded_shard_with_metrics
Errors
prepare_owned_xorb_file_with_checkpoint
Authenticate and normalize owned staging once, preserving its validation result for persistence. Call this synchronous operation under bounded blocking admission and retain ownership until its checkpoint reports stop.
prepare_owned_xorb_memory_with_checkpoint
Prepare an owned native request without staging files or cloning canonical input. The caller must retain memory/CPU admission until this worker stops.
prepare_uploaded_shard
Performs bounded, read-only verification of a native shard and its sources.
prepare_uploaded_shard_with_checkpoint
Read-only preparation with a request-lifetime check between verification units. The callback must return an interrupted I/O error after request cancellation. S3 materialization checks between response frames; an already-waiting read retains the storage backend’s timeout. Local reads retain the source inode.
prepare_uploaded_shard_with_digest_worker
Overlap SHA-256 with source authentication using an independently admitted CPU worker. The caller must supply and retain its admission token; this function releases it only after all digest workers have stopped.
prepare_xorb_memory
Fully validate and normalize a bounded in-memory upload before deriving its identity. Retain this owned result through publication instead of repeating normalization and native-content validation afterwards.
read_verified_xorb_chunk
Reads a decoded member without relying on a standalone chunk copy or trusting derived descriptor offsets. The complete source is verified before any member is returned. Missing containers or members return None; corruption is an error.
reconstruction_v2_from_v1
recover_dedup_xorb_descriptor
Recovers a bounded descriptor for an old chunk-to-shard mapping.
recover_dedup_xorb_descriptor_from_candidates
Fully validates a source selected from previously bounded discovery hints. Missing sources are skipped; source/member validation remains mandatory.
recover_xorb_descriptor
Rebuilds a native descriptor by authenticating its complete immutable Xorb. The result is derived metadata, not proof of repository authorization.
register_uploaded_shard_bytes
Errors
register_uploaded_shard_bytes_with_member_references
Registers a shard and supplies physical member references obtained during the existing xorb validation. No second object read or cache lookup is used. The caller must check these references atomically with metadata publication.
register_uploaded_shard_bytes_with_verified_descriptors
Registers bounded shard metadata with descriptors from the same complete Xorb verification pass. The callback must publish the metadata atomically.
register_uploaded_shard_file
Registers a shard parsed directly from a temporary file. Metadata sections are bounded by shard_metadata_limits; the uploaded byte stream itself is never copied into a process-sized buffer.
register_uploaded_shard_file_with_member_references
File-backed registration with references from the same validation pass. The uploaded body is not copied into a process-sized buffer. The caller must check these references atomically with metadata publication.
register_uploaded_shard_file_with_verified_descriptors
File-backed registration with verified descriptors and protected references.
resolve_dedupe_shard_object
Errors
retained_shard_chunk_hashes
Errors
retained_shard_chunk_hashes_from_reader
Reads retained chunk metadata from a seekable shard without requiring the caller to first materialize the entire shard in memory.
shard_hash_from_object_key_if_present
Errors
shard_object_key
Errors
store_owned_xorb_file
Persists an owned staged container, retaining application admission through any synchronous write that outlives cancellation of its caller.
store_prepared_xorb_file
Persist a prepared owned container without repeating its complete format validation or storing decoded copies. Immutable publication rechecks the serialized digest. Started writes keep staging and admission after caller cancellation. Independent chunk readers require published member discovery metadata and support for authenticated Xorb-backed reads.
store_prepared_xorb_memory
Publish owned prepared bytes while retaining admission through synchronous storage work. Storage checks the bound immutable commitment before publication; conflicting encodings still require complete verification of retained bytes.
store_uploaded_xorb
Errors
store_uploaded_xorb_bytes
Errors
store_uploaded_xorb_file
Stores only the canonical serialized Xorb from a seekable file. Decoded members are authenticated during validation, without a second stored copy. Footerless inputs use verified file-backed normalization with bounded memory.
store_uploaded_xorb_file_path
Errors
store_uploaded_xorb_with_metrics
Errors
store_xorb_bytes_with_resources
Stores verified in-memory input while retaining admission through immutable persistence and any full verification of an equivalent retained encoding.
try_for_each_serialized_xorb_chunk
Decodes the packed chunk stream of a previously validated serialized xorb and passes each decoded chunk to visitor before decoding the next chunk.
try_for_each_serialized_xorb_chunk_async
Decodes the packed chunk stream of a previously validated serialized xorb and passes each decoded chunk to an async visitor before decoding the next chunk.
try_for_each_serialized_xorb_chunk_async_trusted
Async counterpart to try_for_each_serialized_xorb_chunk_trusted.
try_for_each_serialized_xorb_chunk_range
Decodes and rehashes a nonempty member range of a fully validated Xorb.
try_for_each_serialized_xorb_chunk_trusted
Decodes a serialized xorb after the exact immutable bytes have already been fully validated by validate_serialized_xorb.
validate_hash_path
Validates that a hash path is a 64-character lowercase hex digest.
validate_optional_content_hash
Validates an optional content hash, passing when the value is None.
validate_serialized_xorb
Validates a serialized Xet xorb against the requested content hash and extracts trusted chunk metadata.
validate_serialized_xorb_footer
Authenticates a footer against an independently trusted Xorb root. Legacy boundaries return None and require complete-container validation.
validate_serialized_xorb_with_data
Fully validates a container and retains its decoded payload only when it fits within max_decoded_bytes. Invalid containers never return payloads. The supplied buffer is cleared and reused for the result allocation.
validate_xorb_transfer_namespace
Validates the transfer namespace prefix for xorb transfer routes.
visit_stored_xorb_chunk_hashes
Errors
visit_stored_xorb_chunk_hashes_if_present
Visits members only after fully validating their stored container.
xorb_chunks_cache_hash_from_key_if_present
Extracts a Xorb hash from a legacy member-cache sidecar key. Returns None when the key does not match the cache-namespace pattern.
xorb_hash_from_object_key_if_present
Errors
xorb_object_key
Errors