Expand description
HuggingFace Hub API compatibility layer for Shardline.
This crate provides an Axum-based HTTP API that makes Shardline a drop-in
HuggingFace Hub alternative. Users can point huggingface-cli at a Shardline
server and upload/download models as if it were the real Hub.
§Quick start
The commit-parsing helpers are pure and are the easiest place to start: they turn the Hub’s NDJSON commit stream into typed instructions that the rest of the API applies to storage.
use shardline_hub_api::commit::parse_ndjson_commit;
use shardline_hub_api::commit::CommitInstruction;
use base64::Engine;
let content = base64::engine::general_purpose::STANDARD.encode(b"hello world");
let body = format!(
"{{\"header\":{{\"summary\":\"add readme\",\"parentCommit\":\"\"}}}}\n\
{{\"file\":{{\"path\":\"README.md\",\"content\":\"{content}\"}}}}"
);
let commit = parse_ndjson_commit(&body)?;
assert_eq!(commit.message, "add readme");
assert!(matches!(
&commit.instructions[0],
CommitInstruction::InlineFile { path, content } if path == "README.md" && content == b"hello world"
));To serve the API itself, build a HubState backed by a
Shardline index store and object store, then mount hub_routes into your
Axum application:
use axum::Router;
use shardline_hub_api::hub_routes;
use shardline_hub_api::routes::HubState;
let app: Router = hub_routes(state, true);Modules§
- auth
- commit
- error
- git
- Git Smart HTTP protocol implementation.
- models
- query
- Structured contract for an optional isolated dataset query worker.
- resolve
- routes
- secrets
- At-rest encryption for Hub webhook signing secrets.
- state
- types
- Shared value types for the Hub API.
Functions§
- hub_
routes - Builds the Hub API router with all registered routes.
- hub_
routes_ with_ dataset_ query - Builds Hub routes with an explicit native dataset-query feature gate.
Passing
falsekeeps all existing Hub and legacy dataset-preview routes but omits the structured Parquet query endpoint.