1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
//! `sdx` is a native Xet client library for Shardline's Xet frontend.
//!
//! It downloads files from (and uploads files to) Xet repositories through the
//! shardline CAS frontend: the client resolves a repo-scoped read/write token,
//! fetches a file's reconstruction plan, streams and decodes the chunk data,
//! and writes it out — with bounded memory and an optional on-disk chunk
//! cache. Files are addressed by their 64-hex content hash (`file_id`).
//!
//! # Quick start
//!
//! Point [`XetClientBuilder`] at a `xet://` endpoint and an [`Auth`] scope,
//! then download a file by its 64-hex content id:
//!
//! ```no_run
//! use sdx::{Auth, RepositoryId, XetClientBuilder};
//!
//! #[tokio::main]
//! async fn main() -> Result<(), Box<dyn std::error::Error>> {
//! // The credential is resolved from the SHARDLINE_TOKEN /
//! // SHARDLINE_API_KEY / SHARDLINE_TOKEN_FILE environment variables when
//! // the download runs, or set explicitly with `.with_token(..)` /
//! // `.with_api_key(..)`.
//! let auth = Auth::new(
//! "https://xet.example.com",
//! RepositoryId {
//! provider: "github".to_owned(),
//! owner: "shardline".to_owned(),
//! repo: "assets".to_owned(),
//! revision: "main".to_owned(),
//! },
//! )?;
//!
//! let client = XetClientBuilder::new()
//! .endpoint("xet://xet.example.com/github/shardline/assets/main")
//! .auth(auth)
//! .build()?;
//!
//! // `file_id` is the 64-hex content hash of the file to fetch (from a
//! // tree listing, an upload, or a reconstruction response).
//! let file_id =
//! "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
//! let bytes = client.download_bytes(file_id).await?;
//! println!("downloaded {} bytes", bytes.len());
//! Ok(())
//! }
//! ```
//!
//! # Design history (milestones)
//!
//! The milestone notes below (M0–M4, `docs/SDX_PLAN.md`) are the crate's
//! design history and are not needed to use the library:
//!
//! M0 provides the hash and hexadecimal primitives the Xet wire protocol
//! requires: BLAKE3 keyed hashing for chunk data (and term verification) via
//! the pinned upstream `xet-core-structures` crate, plus strict Xet CAS API
//! hexadecimal conversion with 8-byte group reversal.
//!
//! M1 adds the authentication/token service: [`auth::TokenService`] issues and
//! caches repo+revision-scoped read/write CAS tokens against the shardline
//! token-issuance routes (single-flight refresh with a 30-second buffer), with
//! credential resolution from explicit sources, token files, and the
//! `SHARDLINE_TOKEN` / `SHARDLINE_API_KEY` / `SHARDLINE_TOKEN_FILE`
//! environment variables ([`config`]).
//!
//! M2a adds the core read path: [`client::XetClient`] / [`client::XetClientBuilder`]
//! map a `xet://` endpoint to a repository and token service;
//! [`transfer::TransferClient`] issues reconstruction requests and ranged xorb
//! fetches; [`xorb::XorbReader`] decodes serialized xorb byte ranges (footer-less
//! tolerant, via the pinned `xet-core-structures` public API);
//! [`reconstruction::reconstruct`] orchestrates V2→V1 reconstruction, term
//! resolution, `unpacked_length` validation, and byte-range assembly;
//! [`session::DownloadSession`] exposes `download_file` / `download_range`.
//!
//! M2b1 adds the streaming download core ([`stream`], `docs/SDX_PLAN.md`
//! §4.4.1): pull-based [`stream::DownloadStream`] /
//! [`stream::UnorderedDownloadStream`] (`next()`/`blocking_next()`), the
//! byte-denominated [`stream::BufferSemaphore`] memory bound, and
//! `download_stream` / `download_unordered_stream` / `download_to_writer` /
//! `download_bytes` on [`client::XetClient`] and [`session::DownloadSession`].
//!
//! M2b2 adds the on-disk chunk cache ([`cache::ChunkCache`],
//! `docs/SDX_PLAN.md` §4.4.1 step 3) and the stream-group layer
//! ([`group::XetStreamGroup`], §4.4.3). The cache is content-addressed by the
//! 64-hex xorb hash, atomic (temp+rename), LRU budget-bounded (default 2 GiB),
//! and tolerant of corruption (CRC-verified, delete-on-corrupt); every ranged
//! xorb fetch checks the cache before acquiring the download permit / hitting
//! the network. The stream group manages concurrent downloads with abort-all
//! and per-stream [`group::XetTaskState`] status probes; the upload/commit
//! side of the group layer is M3 and not implemented.
//!
//! M3a adds the offline-testable write-path foundation (all of `docs/SDX_PLAN.md`
//! §4.4.2 / §9-M3): [`chunker::Chunker`] is a streaming CDC chunker
//! **byte-identical** to the server's `CdcChunker`; [`xorb_build::build_xorb`]
//! produces serialized xorbs **byte-identical** to the server's
//! `pack_chunks_into_xorb` (via the pinned upstream `xet-core-structures`
//! crate, with the shardline format-v2 footer assembled directly — see the
//! module docs); [`dedup::DedupClient`] implements the global dedup query
//! (`GET /v1/chunks/default-merkledb/{hash}`, 404 = miss, 429 no-retry) plus
//! the eligibility and defrag-prevention hysteresis helpers.
//!
//! M3b adds the network + session layer on top: [`upload::UploadSession`]
//! runs the push-style ingest loop (8 MiB blocks, CDC on a compute thread,
//! session + eligibility-gated global dedup, HEAD-probed idempotent xorb
//! uploads with streaming bodies from a `JoinSet`, xorbs-before-shard via
//! fork-format v3 shards serialized by [`shard`]) and the streaming surface
//! ([`upload::UploadStreamHandle`], [`crate::XetClient::upload_file`] /
//! `upload_bytes` / `upload_stream`, and the group layer
//! [`group::XetUploadCommit`]). Retry/backoff/adaptive concurrency is M4.
//!
//! Later milestones add retry/adaptive concurrency (M4), chunking, shard, and
//! path addressing modules from the module map in `docs/SDX_PLAN.md` §4.2.
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use RetryPolicy;
pub use Revision;
pub use DownloadSession;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use XetUrl;
pub use MerkleHash;
pub use ;
pub use ;