Skip to main content

Crate shardline_hub_api

Crate shardline_hub_api 

Source
Expand description

HuggingFace Hub API compatibility layer for Shardline.

This crate provides an Axum-based HTTP API that makes Shardline a drop-in HuggingFace Hub alternative. Users can point huggingface-cli at a Shardline server and upload/download models as if it were the real Hub.

§Quick start

The commit-parsing helpers are pure and are the easiest place to start: they turn the Hub’s NDJSON commit stream into typed instructions that the rest of the API applies to storage.

use shardline_hub_api::commit::parse_ndjson_commit;
use shardline_hub_api::commit::CommitInstruction;
use base64::Engine;

let content = base64::engine::general_purpose::STANDARD.encode(b"hello world");
let body = format!(
    "{{\"header\":{{\"summary\":\"add readme\",\"parentCommit\":\"\"}}}}\n\
     {{\"file\":{{\"path\":\"README.md\",\"content\":\"{content}\"}}}}"
);

let commit = parse_ndjson_commit(&body)?;
assert_eq!(commit.message, "add readme");
assert!(matches!(
    &commit.instructions[0],
    CommitInstruction::InlineFile { path, content } if path == "README.md" && content == b"hello world"
));

To serve the API itself, build a HubState backed by a Shardline index store and object store, then mount hub_routes into your Axum application:

use axum::Router;
use shardline_hub_api::hub_routes;
use shardline_hub_api::routes::HubState;

let app: Router = hub_routes(state, true);

Modules§

auth
commit
error
git
Git Smart HTTP protocol implementation.
models
query
Structured contract for an optional isolated dataset query worker.
resolve
routes
secrets
At-rest encryption for Hub webhook signing secrets.
state
types
Shared value types for the Hub API.

Functions§

hub_routes
Builds the Hub API router with all registered routes.
hub_routes_with_dataset_query
Builds Hub routes with an explicit native dataset-query feature gate. Passing false keeps all existing Hub and legacy dataset-preview routes but omits the structured Parquet query endpoint.