Struct HfPublisher 

pub struct HfPublisher { /* private fields */ }

Publisher for uploading datasets to HuggingFace Hub.

Example

use alimentar::hf_hub::HfPublisher;
use arrow::record_batch::RecordBatch;

let publisher = HfPublisher::new("paiml/my-dataset")
    .with_token(std::env::var("HF_TOKEN").unwrap())
    .with_private(false);

// publisher.upload_parquet("train.parquet", &batch).unwrap();

Implementations

impl HfPublisher

pub fn new(repo_id: impl Into<String>) -> Self

Creates a new publisher for a HuggingFace dataset repository.

pub fn with_token(self, token: impl Into<String>) -> Self

Sets the HuggingFace API token.

pub fn with_private(self, private: bool) -> Self

Sets whether the dataset should be private.

pub fn with_commit_message(self, message: impl Into<String>) -> Self

Sets the commit message for uploads.

pub fn repo_id(&self) -> &str

Returns the repository ID.

pub async fn create_repo(&self) -> Result<()>

Creates the dataset repository on HuggingFace Hub if it doesn’t exist.

pub async fn upload_file(&self, path_in_repo: &str, data: &[u8]) -> Result<()>

Uploads a file to the repository.

This method automatically selects the appropriate upload method:

• Binary files (parquet, images, etc.): Uses LFS preupload API
• Text files (README.md, JSON, etc.): Uses direct NDJSON commit API

The official hf-hub crate only supports downloads, making this upload capability a key differentiator for alimentar.

pub async fn upload_batch( &self, path_in_repo: &str, batch: &RecordBatch, ) -> Result<()>

Uploads a RecordBatch as a parquet file.

pub async fn upload_parquet_file( &self, local_path: &Path, path_in_repo: &str, ) -> Result<()>

Uploads a local parquet file to the repository.

pub fn create_repo_sync(&self) -> Result<()>

Synchronous wrapper for creating repo (for CLI use).

pub fn upload_file_sync(&self, path_in_repo: &str, data: &[u8]) -> Result<()>

Synchronous wrapper for uploading file (for CLI use).

pub fn upload_parquet_file_sync( &self, local_path: &Path, path_in_repo: &str, ) -> Result<()>

Synchronous wrapper for uploading parquet file (for CLI use).

pub async fn upload_readme_validated(&self, content: &str) -> Result<()>

Uploads a README.md with validation.

Validates the dataset card metadata before upload to catch issues like invalid task_categories before they cause HuggingFace warnings.

Errors

Returns an error if validation fails or upload fails.

pub fn upload_readme_validated_sync(&self, content: &str) -> Result<()>

Synchronous wrapper for validated README upload.

Trait Implementations

impl Clone for HfPublisher

fn clone(&self) -> HfPublisher

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HfPublisher

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.