xet-data 1.5.1

Data processing pipeline for chunking, deduplication, and file reconstruction; used in the Hugging Face Xet client tools. Intended to be used through the API in the hf-xet package.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

# xet-data

[![crates.io](https://img.shields.io/crates/v/xet-data.svg)](https://crates.io/crates/xet-data)
[![docs.rs](https://docs.rs/xet-data/badge.svg)](https://docs.rs/xet-data)
[![License](https://img.shields.io/crates/l/xet-data.svg)](https://github.com/huggingface/xet-core/blob/main/LICENSE)

Data processing pipeline for chunking, deduplication, and file reconstruction.  Intended to be used through the API in the hf-xet package. 

## Overview

- **Content-defined chunking** — Gear-hash based chunking for deduplication
- **Deduplication** — Probe and register chunks against metadata shards
- **File reconstruction** — Reassemble files from deduplicated chunk references
- **Progress tracking** — Hooks for upload/download progress reporting

This crate is part of [xet-core](https://github.com/huggingface/xet-core).

## License

Apache-2.0