Skip to main content

Crate xet

Crate xet 

Source
Expand description

Rust client library for the Hugging Face Xet storage system.

Xet is the storage backend used by the Hugging Face Hub for large files. Files are split into variable-size chunks, deduplicated, and stored in CAS (Content-Addressed Storage) server. This crate provides a high-level API for uploading and downloading those files.

§Getting started

All operations go through xet_session::XetSession, which manages a tokio runtime and shared HTTP settings. Create one with XetSessionBuilder, then use it to build upload commits or download groups:

use xet::xet_session::{Sha256Policy, XetFileInfo, XetSessionBuilder};

let session = XetSessionBuilder::new().build()?;

// Upload a file
let commit = session
    .new_upload_commit()?
    .with_endpoint("https://cas.example.com")
    .with_token_info("write-token", 1_700_000_000)
    .build_blocking()?;
let handle = commit.upload_from_path_blocking("file.bin".into(), Sha256Policy::Compute)?;
let report = commit.commit_blocking()?;

// Download a file using the metadata from the upload
let meta = report.uploads.values().next().unwrap();
let group = session
    .new_file_download_group()?
    .with_token_info("read-token", 1_700_000_000)
    .build_blocking()?;
group.download_file_to_path_blocking(meta.xet_info.clone(), "out/file.bin".into())?;
group.finish_blocking()?;

See the xet_session module for the full API, including async variants, streaming uploads and downloads, and progress tracking.

§WebAssembly targets

This crate compiles for wasm32-unknown-unknown, but the public surface is a strict subset of the native API. The example above uses _blocking variants and upload_from_path / download_file_to_path, all of which are non-wasm-only — wasm cannot block the host thread and has no filesystem. On wasm the supported entry points are the async variants of new_upload_commit (with upload_bytes / upload_stream) and new_download_stream_group (with download_stream). The legacy module and XetSession::new_file_download_group (along with all _blocking variants) are non-wasm-only.

§Modules

  • xet_session — the primary API: XetSession, upload commits, file download groups, and streaming downloads.
  • error — XetError, the unified error type for the public API.

Re-exports§

pub use error::XetError;

Modules§

error
legacy
Legacy helpers re-exported for backward compatibility.
xet_session
Session-based file upload and download API for XetHub / HuggingFace Hub.

Functions§

init_logging
Initialize the global tracing subscriber using xet_runtime defaults.