Skip to main content

Crate sdx

Crate sdx 

Source
Expand description

sdx is a native Xet client library for Shardline’s Xet frontend.

It downloads files from (and uploads files to) Xet repositories through the shardline CAS frontend: the client resolves a repo-scoped read/write token, fetches a file’s reconstruction plan, streams and decodes the chunk data, and writes it out — with bounded memory and an optional on-disk chunk cache. Files are addressed by their 64-hex content hash (file_id).

§Quick start

Point XetClientBuilder at a xet:// endpoint and an Auth scope, then download a file by its 64-hex content id:

use sdx::{Auth, RepositoryId, XetClientBuilder};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // The credential is resolved from the SHARDLINE_TOKEN /
    // SHARDLINE_API_KEY / SHARDLINE_TOKEN_FILE environment variables when
    // the download runs, or set explicitly with `.with_token(..)` /
    // `.with_api_key(..)`.
    let auth = Auth::new(
        "https://xet.example.com",
        RepositoryId {
            provider: "github".to_owned(),
            owner: "shardline".to_owned(),
            repo: "assets".to_owned(),
            revision: "main".to_owned(),
        },
    )?;

    let client = XetClientBuilder::new()
        .endpoint("xet://xet.example.com/github/shardline/assets/main")
        .auth(auth)
        .build()?;

    // `file_id` is the 64-hex content hash of the file to fetch (from a
    // tree listing, an upload, or a reconstruction response).
    let file_id =
        "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
    let bytes = client.download_bytes(file_id).await?;
    println!("downloaded {} bytes", bytes.len());
    Ok(())
}

§Design history (milestones)

The milestone notes below (M0–M4, docs/SDX_PLAN.md) are the crate’s design history and are not needed to use the library:

M0 provides the hash and hexadecimal primitives the Xet wire protocol requires: BLAKE3 keyed hashing for chunk data (and term verification) via the pinned upstream xet-core-structures crate, plus strict Xet CAS API hexadecimal conversion with 8-byte group reversal.

M1 adds the authentication/token service: auth::TokenService issues and caches repo+revision-scoped read/write CAS tokens against the shardline token-issuance routes (single-flight refresh with a 30-second buffer), with credential resolution from explicit sources, token files, and the SHARDLINE_TOKEN / SHARDLINE_API_KEY / SHARDLINE_TOKEN_FILE environment variables (config).

M2a adds the core read path: client::XetClient / client::XetClientBuilder map a xet:// endpoint to a repository and token service; transfer::TransferClient issues reconstruction requests and ranged xorb fetches; xorb::XorbReader decodes serialized xorb byte ranges (footer-less tolerant, via the pinned xet-core-structures public API); reconstruction::reconstruct orchestrates V2→V1 reconstruction, term resolution, unpacked_length validation, and byte-range assembly; session::DownloadSession exposes download_file / download_range.

M2b1 adds the streaming download core (stream, docs/SDX_PLAN.md §4.4.1): pull-based stream::DownloadStream / stream::UnorderedDownloadStream (next()/blocking_next()), the byte-denominated stream::BufferSemaphore memory bound, and download_stream / download_unordered_stream / download_to_writer / download_bytes on client::XetClient and session::DownloadSession.

M2b2 adds the on-disk chunk cache (cache::ChunkCache, docs/SDX_PLAN.md §4.4.1 step 3) and the stream-group layer (group::XetStreamGroup, §4.4.3). The cache is content-addressed by the 64-hex xorb hash, atomic (temp+rename), LRU budget-bounded (default 2 GiB), and tolerant of corruption (CRC-verified, delete-on-corrupt); every ranged xorb fetch checks the cache before acquiring the download permit / hitting the network. The stream group manages concurrent downloads with abort-all and per-stream group::XetTaskState status probes; the upload/commit side of the group layer is M3 and not implemented.

M3a adds the offline-testable write-path foundation (all of docs/SDX_PLAN.md §4.4.2 / §9-M3): chunker::Chunker is a streaming CDC chunker byte-identical to the server’s CdcChunker; xorb_build::build_xorb produces serialized xorbs byte-identical to the server’s pack_chunks_into_xorb (via the pinned upstream xet-core-structures crate, with the shardline format-v2 footer assembled directly — see the module docs); dedup::DedupClient implements the global dedup query (GET /v1/chunks/default-merkledb/{hash}, 404 = miss, 429 no-retry) plus the eligibility and defrag-prevention hysteresis helpers.

M3b adds the network + session layer on top: upload::UploadSession runs the push-style ingest loop (8 MiB blocks, CDC on a compute thread, session + eligibility-gated global dedup, HEAD-probed idempotent xorb uploads with streaming bodies from a JoinSet, xorbs-before-shard via fork-format v3 shards serialized by shard) and the streaming surface (upload::UploadStreamHandle, crate::XetClient::upload_file / upload_bytes / upload_stream, and the group layer group::XetUploadCommit). Retry/backoff/adaptive concurrency is M4.

Later milestones add retry/adaptive concurrency (M4), chunking, shard, and path addressing modules from the module map in docs/SDX_PLAN.md §4.2.

Re-exports§

pub use auth::Auth;
pub use auth::AuthError;
pub use auth::HttpConfig;
pub use auth::PROVIDER_KEY_HEADER_NAME;
pub use auth::REFRESH_BUFFER_SECONDS;
pub use auth::RepositoryId;
pub use auth::ScopedToken;
pub use auth::TokenService;
pub use cache::CachedXorbRange;
pub use cache::ChunkCache;
pub use cache::DEFAULT_CHUNK_CACHE_BUDGET_BYTES;
pub use chunker::Chunk;
pub use chunker::Chunker;
pub use chunker::DEFAULT_TARGET_CHUNK_SIZE;
pub use client::XetClient;
pub use client::XetClientBuilder;
pub use config::AuthSection;
pub use config::Credential;
pub use config::DefaultSection;
pub use config::SdxConfig;
pub use config::SdxConfigError;
pub use config::resolve_credential_from_config;
pub use config::resolve_credential_from_env;
pub use dedup::DEFRAG_HYSTERESIS_FACTOR;
pub use dedup::DEFRAG_MIN_CHUNKS_PER_RANGE;
pub use dedup::DEFRAG_NUM_RANGES_WINDOW;
pub use dedup::DedupClient;
pub use dedup::DedupOutcome;
pub use dedup::DefragPrevention;
pub use dedup::GLOBAL_DEDUP_CHUNK_MODULUS;
pub use dedup::GLOBAL_DEDUP_PREFIX;
pub use dedup::MIN_SPACING_BETWEEN_GLOBAL_DEDUP_QUERIES;
pub use dedup::is_global_dedup_eligible;
pub use error::SdxError;
pub use error::TransferError;
pub use error::XetHashParseError;
pub use group::GroupedDownloadStream;
pub use group::GroupedUnorderedDownloadStream;
pub use group::GroupedUploadStream;
pub use group::XetStreamGroup;
pub use group::XetTaskState;
pub use group::XetUploadCommit;
pub use hash::compute_chunk_hash;
pub use hash::compute_term_verification_hash;
pub use hash::parse_xet_hash_hex;
pub use hash::xet_hash_hex_string;
pub use reconstruction::ReconstructedFile;
pub use reconstruction::ResolvedTerm;
pub use reconstruction::reconstruct;
pub use retry::RetryPolicy;
pub use revisions::Revision;
pub use session::DownloadSession;
pub use shard::ShardFileEntry;
pub use shard::ShardSegment;
pub use shard::ShardXorb;
pub use shard::ShardXorbChunk;
pub use shard::serialize_shard;
pub use stream::BufferPermit;
pub use stream::BufferSemaphore;
pub use stream::DEFAULT_COMPLETION_RATE_ESTIMATOR_HALF_LIFE;
pub use stream::DEFAULT_DOWNLOAD_BUFFER_LIMIT;
pub use stream::DEFAULT_DOWNLOAD_BUFFER_PERFILE_SIZE;
pub use stream::DEFAULT_DOWNLOAD_BUFFER_SIZE;
pub use stream::DEFAULT_DOWNLOAD_CONCURRENCY;
pub use stream::DEFAULT_MAX_RECONSTRUCTION_FETCH_SIZE;
pub use stream::DEFAULT_MIN_PREFETCH_BUFFER;
pub use stream::DEFAULT_MIN_RECONSTRUCTION_FETCH_SIZE;
pub use stream::DEFAULT_TARGET_BLOCK_COMPLETION_TIME_SECS;
pub use stream::DataFuture;
pub use stream::DataWriter;
pub use stream::DownloadStream;
pub use stream::StreamLimits;
pub use stream::UnorderedDownloadStream;
pub use transfer::ByteRange;
pub use transfer::MultipartPart;
pub use transfer::RangedXorb;
pub use transfer::TransferClient;
pub use transfer::XORB_UPLOAD_PROGRESS_BLOCK_SIZE;
pub use transfer::parse_multipart_byteranges;
pub use tree::DirEntry;
pub use tree::DirListing;
pub use tree::PathEntry;
pub use tree::RegisterResult;
pub use upload::DEFAULT_UPLOAD_CONCURRENCY;
pub use upload::INGESTION_BLOCK_SIZE;
pub use upload::UploadFileInfo;
pub use upload::UploadReport;
pub use upload::UploadSession;
pub use upload::UploadStreamHandle;
pub use url::XetUrl;
pub use xorb::DecodedChunk;
pub use xorb::XorbError;
pub use xorb::XorbReader;
pub use xorb_build::BuiltXorb;
pub use xorb_build::MAX_XORB_BYTES;
pub use xorb_build::MAX_XORB_CHUNKS;
pub use xorb_build::SERIALIZED_XORB_SAFETY_CAP_BYTES;
pub use xorb_build::XorbChunkEntry;
pub use xorb_build::build_xorb;
pub use xorb_build::serialized_size_le;
pub use xorb_build::xorb_cut_condition;
pub use xorb_build::xorb_max_addable_chunk;

Modules§

auth
Authentication for Xet repositories: issue, cache, and transparently refresh repo+revision-scoped read/write CAS tokens.
cache
On-disk chunk cache (M2b2, docs/SDX_PLAN.md §4.4.1 step 3).
chunker
Content-defined chunking (M3a): the client CDC chunker.
client
Build and use a Xet client: download files from (and upload files to) a Xet repository, addressed by 64-hex content id.
config
Credential resolution for the sdx token service: from environment variables, token files, and the shardline.toml config file.
dedup
Global dedup query + eligibility (M3a).
error
group
Stream-group layer (M2b2, docs/SDX_PLAN.md §4.4.3).
hash
BLAKE3 keyed hashing and Xet CAS API hexadecimal conversion.
reconstruction
Reconstruction orchestration for the sdx CAS read path (M2a).
retry
Retry / backoff / token-refresh transport behavior (M4, docs/SDX_PLAN.md §4.4.4 + §6.3).
revisions
Revision/branch client (M5b, docs/SDX_PLAN.md §4.3): create, list, and delete revisions against the M5a revision-registry endpoints.
session
Download sessions for the sdx CAS read path (M2a / M2b1).
shard
Serialize and parse Xet metadata shards — the manifest that maps files to the xorb chunk ranges that contain their data.
stream
Pull-based streaming download core (M2b1, docs/SDX_PLAN.md §4.4.1).
transfer
HTTP client for the Xet CAS data plane: file-reconstruction requests and ranged xorb (chunk-container) fetches.
tree
Path-namespace client (M5b, docs/SDX_PLAN.md §4.3): map paths to content file_ids and back, against the M5a server metadata endpoints.
upload
Upload session layer (M3b, docs/SDX_PLAN.md §4.4.2 / §9-M3).
url
Full xet:// URL parsing: repository identity plus an optional path.
xorb
Parsing and decoding of serialized Xorb byte ranges (M2a).
xorb_build
Xorb build / serialize primitives (M3a).

Type Aliases§

MerkleHash