xet-core-structures

Core data structures for the Hugging Face Xet storage system, including Merkle hashes, metadata shards, and Xorb objects.

Overview

• MerkleHash — 256-bit content-addressed hash used throughout the system
• Metadata shards — Compact shard format mapping file ranges to Xorb chunks
• Xorb objects — Content-addressed storage objects with byte-grouping compression
• Data structures — Specialized hash maps and utilities for deduplication

This crate is part of xet-core, the Rust backend for huggingface_hub.

License

Apache-2.0