Expand description
sdx is a native Xet client library for Shardline’s Xet frontend.
It downloads files from (and uploads files to) Xet repositories through the
shardline CAS frontend: the client resolves a repo-scoped read/write token,
fetches a file’s reconstruction plan, streams and decodes the chunk data,
and writes it out — with bounded memory and an optional on-disk chunk
cache. Files are addressed by their 64-hex content hash (file_id).
§Quick start
Point XetClientBuilder at a xet:// endpoint and an Auth scope,
then download a file by its 64-hex content id:
use sdx::{Auth, RepositoryId, XetClientBuilder};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// The credential is resolved from the SHARDLINE_TOKEN /
// SHARDLINE_API_KEY / SHARDLINE_TOKEN_FILE environment variables when
// the download runs, or set explicitly with `.with_token(..)` /
// `.with_api_key(..)`.
let auth = Auth::new(
"https://xet.example.com",
RepositoryId {
provider: "github".to_owned(),
owner: "shardline".to_owned(),
repo: "assets".to_owned(),
revision: "main".to_owned(),
},
)?;
let client = XetClientBuilder::new()
.endpoint("xet://xet.example.com/github/shardline/assets/main")
.auth(auth)
.build()?;
// `file_id` is the 64-hex content hash of the file to fetch (from a
// tree listing, an upload, or a reconstruction response).
let file_id =
"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
let bytes = client.download_bytes(file_id).await?;
println!("downloaded {} bytes", bytes.len());
Ok(())
}§Design history (milestones)
The milestone notes below (M0–M4, docs/SDX_PLAN.md) are the crate’s
design history and are not needed to use the library:
M0 provides the hash and hexadecimal primitives the Xet wire protocol
requires: BLAKE3 keyed hashing for chunk data (and term verification) via
the pinned upstream xet-core-structures crate, plus strict Xet CAS API
hexadecimal conversion with 8-byte group reversal.
M1 adds the authentication/token service: auth::TokenService issues and
caches repo+revision-scoped read/write CAS tokens against the shardline
token-issuance routes (single-flight refresh with a 30-second buffer), with
credential resolution from explicit sources, token files, and the
SHARDLINE_TOKEN / SHARDLINE_API_KEY / SHARDLINE_TOKEN_FILE
environment variables (config).
M2a adds the core read path: client::XetClient / client::XetClientBuilder
map a xet:// endpoint to a repository and token service;
transfer::TransferClient issues reconstruction requests and ranged xorb
fetches; xorb::XorbReader decodes serialized xorb byte ranges (footer-less
tolerant, via the pinned xet-core-structures public API);
reconstruction::reconstruct orchestrates V2→V1 reconstruction, term
resolution, unpacked_length validation, and byte-range assembly;
session::DownloadSession exposes download_file / download_range.
M2b1 adds the streaming download core (stream, docs/SDX_PLAN.md
§4.4.1): pull-based stream::DownloadStream /
stream::UnorderedDownloadStream (next()/blocking_next()), the
byte-denominated stream::BufferSemaphore memory bound, and
download_stream / download_unordered_stream / download_to_writer /
download_bytes on client::XetClient and session::DownloadSession.
M2b2 adds the on-disk chunk cache (cache::ChunkCache,
docs/SDX_PLAN.md §4.4.1 step 3) and the stream-group layer
(group::XetStreamGroup, §4.4.3). The cache is content-addressed by the
64-hex xorb hash, atomic (temp+rename), LRU budget-bounded (default 2 GiB),
and tolerant of corruption (CRC-verified, delete-on-corrupt); every ranged
xorb fetch checks the cache before acquiring the download permit / hitting
the network. The stream group manages concurrent downloads with abort-all
and per-stream group::XetTaskState status probes; the upload/commit
side of the group layer is M3 and not implemented.
M3a adds the offline-testable write-path foundation (all of docs/SDX_PLAN.md
§4.4.2 / §9-M3): chunker::Chunker is a streaming CDC chunker
byte-identical to the server’s CdcChunker; xorb_build::build_xorb
produces serialized xorbs byte-identical to the server’s
pack_chunks_into_xorb (via the pinned upstream xet-core-structures
crate, with the shardline format-v2 footer assembled directly — see the
module docs); dedup::DedupClient implements the global dedup query
(GET /v1/chunks/default-merkledb/{hash}, 404 = miss, 429 no-retry) plus
the eligibility and defrag-prevention hysteresis helpers.
M3b adds the network + session layer on top: upload::UploadSession
runs the push-style ingest loop (8 MiB blocks, CDC on a compute thread,
session + eligibility-gated global dedup, HEAD-probed idempotent xorb
uploads with streaming bodies from a JoinSet, xorbs-before-shard via
fork-format v3 shards serialized by shard) and the streaming surface
(upload::UploadStreamHandle, crate::XetClient::upload_file /
upload_bytes / upload_stream, and the group layer
group::XetUploadCommit). Retry/backoff/adaptive concurrency is M4.
Later milestones add retry/adaptive concurrency (M4), chunking, shard, and
path addressing modules from the module map in docs/SDX_PLAN.md §4.2.
Re-exports§
pub use auth::Auth;pub use auth::AuthError;pub use auth::HttpConfig;pub use auth::PROVIDER_KEY_HEADER_NAME;pub use auth::REFRESH_BUFFER_SECONDS;pub use auth::RepositoryId;pub use auth::ScopedToken;pub use auth::TokenService;pub use cache::CachedXorbRange;pub use cache::ChunkCache;pub use cache::DEFAULT_CHUNK_CACHE_BUDGET_BYTES;pub use chunker::Chunk;pub use chunker::Chunker;pub use chunker::DEFAULT_TARGET_CHUNK_SIZE;pub use client::XetClient;pub use client::XetClientBuilder;pub use config::AuthSection;pub use config::Credential;pub use config::DefaultSection;pub use config::SdxConfig;pub use config::SdxConfigError;pub use config::resolve_credential_from_config;pub use config::resolve_credential_from_env;pub use dedup::DEFRAG_HYSTERESIS_FACTOR;pub use dedup::DEFRAG_MIN_CHUNKS_PER_RANGE;pub use dedup::DEFRAG_NUM_RANGES_WINDOW;pub use dedup::DedupClient;pub use dedup::DedupOutcome;pub use dedup::DefragPrevention;pub use dedup::GLOBAL_DEDUP_CHUNK_MODULUS;pub use dedup::GLOBAL_DEDUP_PREFIX;pub use dedup::MIN_SPACING_BETWEEN_GLOBAL_DEDUP_QUERIES;pub use dedup::is_global_dedup_eligible;pub use error::SdxError;pub use error::TransferError;pub use error::XetHashParseError;pub use group::GroupedDownloadStream;pub use group::GroupedUnorderedDownloadStream;pub use group::GroupedUploadStream;pub use group::XetStreamGroup;pub use group::XetTaskState;pub use group::XetUploadCommit;pub use hash::compute_chunk_hash;pub use hash::compute_term_verification_hash;pub use hash::parse_xet_hash_hex;pub use hash::xet_hash_hex_string;pub use reconstruction::ReconstructedFile;pub use reconstruction::ResolvedTerm;pub use reconstruction::reconstruct;pub use retry::RetryPolicy;pub use revisions::Revision;pub use session::DownloadSession;pub use shard::ShardFileEntry;pub use shard::ShardSegment;pub use shard::ShardXorb;pub use shard::ShardXorbChunk;pub use shard::serialize_shard;pub use stream::BufferPermit;pub use stream::BufferSemaphore;pub use stream::DEFAULT_COMPLETION_RATE_ESTIMATOR_HALF_LIFE;pub use stream::DEFAULT_DOWNLOAD_BUFFER_LIMIT;pub use stream::DEFAULT_DOWNLOAD_BUFFER_PERFILE_SIZE;pub use stream::DEFAULT_DOWNLOAD_BUFFER_SIZE;pub use stream::DEFAULT_DOWNLOAD_CONCURRENCY;pub use stream::DEFAULT_MAX_RECONSTRUCTION_FETCH_SIZE;pub use stream::DEFAULT_MIN_PREFETCH_BUFFER;pub use stream::DEFAULT_MIN_RECONSTRUCTION_FETCH_SIZE;pub use stream::DEFAULT_TARGET_BLOCK_COMPLETION_TIME_SECS;pub use stream::DataFuture;pub use stream::DataWriter;pub use stream::DownloadStream;pub use stream::StreamLimits;pub use stream::UnorderedDownloadStream;pub use transfer::ByteRange;pub use transfer::MultipartPart;pub use transfer::RangedXorb;pub use transfer::TransferClient;pub use transfer::XORB_UPLOAD_PROGRESS_BLOCK_SIZE;pub use transfer::parse_multipart_byteranges;pub use tree::DirEntry;pub use tree::DirListing;pub use tree::PathEntry;pub use tree::RegisterResult;pub use upload::DEFAULT_UPLOAD_CONCURRENCY;pub use upload::INGESTION_BLOCK_SIZE;pub use upload::UploadFileInfo;pub use upload::UploadReport;pub use upload::UploadSession;pub use upload::UploadStreamHandle;pub use url::XetUrl;pub use xorb::DecodedChunk;pub use xorb::XorbError;pub use xorb::XorbReader;pub use xorb_build::BuiltXorb;pub use xorb_build::MAX_XORB_BYTES;pub use xorb_build::MAX_XORB_CHUNKS;pub use xorb_build::SERIALIZED_XORB_SAFETY_CAP_BYTES;pub use xorb_build::XorbChunkEntry;pub use xorb_build::build_xorb;pub use xorb_build::serialized_size_le;pub use xorb_build::xorb_cut_condition;pub use xorb_build::xorb_max_addable_chunk;
Modules§
- auth
- Authentication for Xet repositories: issue, cache, and transparently refresh repo+revision-scoped read/write CAS tokens.
- cache
- On-disk chunk cache (M2b2,
docs/SDX_PLAN.md§4.4.1 step 3). - chunker
- Content-defined chunking (M3a): the client CDC chunker.
- client
- Build and use a Xet client: download files from (and upload files to) a Xet repository, addressed by 64-hex content id.
- config
- Credential resolution for the
sdxtoken service: from environment variables, token files, and theshardline.tomlconfig file. - dedup
- Global dedup query + eligibility (M3a).
- error
- group
- Stream-group layer (M2b2,
docs/SDX_PLAN.md§4.4.3). - hash
- BLAKE3 keyed hashing and Xet CAS API hexadecimal conversion.
- reconstruction
- Reconstruction orchestration for the sdx CAS read path (M2a).
- retry
- Retry / backoff / token-refresh transport behavior (M4,
docs/SDX_PLAN.md§4.4.4 + §6.3). - revisions
- Revision/branch client (M5b,
docs/SDX_PLAN.md§4.3): create, list, and delete revisions against the M5a revision-registry endpoints. - session
- Download sessions for the sdx CAS read path (M2a / M2b1).
- shard
- Serialize and parse Xet metadata shards — the manifest that maps files to the xorb chunk ranges that contain their data.
- stream
- Pull-based streaming download core (M2b1,
docs/SDX_PLAN.md§4.4.1). - transfer
- HTTP client for the Xet CAS data plane: file-reconstruction requests and ranged xorb (chunk-container) fetches.
- tree
- Path-namespace client (M5b,
docs/SDX_PLAN.md§4.3): map paths to contentfile_ids and back, against the M5a server metadata endpoints. - upload
- Upload session layer (M3b,
docs/SDX_PLAN.md§4.4.2 / §9-M3). - url
- Full
xet://URL parsing: repository identity plus an optional path. - xorb
- Parsing and decoding of serialized Xorb byte ranges (M2a).
- xorb_
build - Xorb build / serialize primitives (M3a).