hf-xet
Client library for the Hugging Face Xet data storage system. Provides the high-level session API for uploading and downloading files with chunk-based deduplication.
Overview
- XetSession — Top-level session managing authentication, configuration, and concurrent file transfers
- Upload & download — Stream files to/from Hugging Face Hub with automatic chunking, deduplication, and local caching
Crate Ecosystem
hf-xet ties together the lower-level xet-core crates:
| Crate | Role |
|---|---|
xet-runtime |
Async runtime, config, logging |
xet-core-structures |
Merkle hashes, shards, Xorb objects |
xet-client |
HTTP client for CAS and Hub APIs |
xet-data |
Chunking, dedup, file reconstruction |
This crate is part of xet-core, the Rust backend for huggingface_hub.
License
Apache-2.0