xet-data 1.5.2

Data processing pipeline for chunking, deduplication, and file reconstruction; used in the Hugging Face Xet client tools. Intended to be used through the API in the hf-xet package.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

//! Data processing pipeline for chunking, deduplication, and file
//! reconstruction, used in the Hugging Face Xet storage tools.
//!
//! Provides content-defined chunking via gear hashing, deduplication
//! against metadata shards, and file reconstruction from deduplicated
//! chunk references.

#![cfg_attr(feature = "strict", deny(warnings))]

pub mod error;
pub use error::{DataError, Result};

pub mod deduplication;
#[cfg(not(target_family = "wasm"))]
pub mod file_reconstruction;
#[cfg(not(target_family = "wasm"))]
pub mod processing;
pub mod progress_tracking;