Shardline

The open-source storage backend for large binary assets.

Shardline is a self-hostable content-addressed storage (CAS) server. It accepts immutable object uploads, deduplicates content, and serves range-aware downloads. Run it standalone or pair it with GitHub, GitLab, or Gitea for repository-scoped storage.

The first open-source, production-ready, multi-protocol CAS server with a Xet frontend. Other Xet-compatible implementations exist as proof-of-concepts or single-protocol servers. Shardline combines Xet with Git LFS, Bazel HTTP remote cache, and OCI Distribution in a self-hostable, Kubernetes-ready deployment with full operational tooling.

What it does

• Store and deduplicate any binary content — datasets, model weights, build artifacts, media
• Multiple protocols — Xet (default), Git LFS, Bazel HTTP remote cache, OCI Distribution
• HuggingFace Hub API — drop-in alternative for huggingface-cli uploads and downloads
• Pluggable auth — local Ed25519, OIDC, JWKS, or passthrough provider adapters
• Self-hosted or cloud — local filesystem, S3-compatible storage, Postgres metadata
• Production-ready — health checks, migrations, integrity verification, garbage collection, backups
• Provider integration — optional webhooks and token issuance for GitHub, GitLab, Gitea, Codeberg

Quick start

# Run with Docker
docker compose -f docker-compose.yml up --build

# Or build from source
cargo build --release
./target/release/shardline serve

The server bootstraps .shardline/ automatically on first run.

For providerless setup without starting the server:

shardline providerless setup

Mint repository-scoped tokens:

shardline admin token

Deployment options

All profiles run providerless by default. Provider integration is optional.

Documentation

License

Dual licensed under MIT or Apache 2.0.