alimentar 0.2.7

Data Loading, Distribution and Tooling in Pure Rust
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122

# HuggingFace Hub (Examples 76-85)

This section covers HuggingFace Hub integration.

## Examples 76-77: Dataset Download and Card Validation

```rust
use alimentar::hf_hub::HfDataset;

// Download dataset
let dataset = HfDataset::builder("username/dataset")
    .revision("main")
    .split("train")
    .build()?
    .download()?;

// With card validation
let hf = HfDataset::builder("username/dataset")
    .validate_card(true)
    .build()?;
let validation = hf.validate_dataset_card()?;
```

## Examples 78-79: Quality Score and README Generation

```rust
use alimentar::hf_hub::{HfDataset, DatasetCardValidator};

let hf = HfDataset::builder("username/dataset").build()?;

// Get quality score
let quality = hf.compute_quality_score()?;
println!("Hub quality: {:.2}", quality);

// Generate README
let validator = DatasetCardValidator::new();
let readme = validator.generate_readme(&dataset)?;
```

## Examples 80-81: Native Upload and Revision

```rust
use alimentar::hf_hub::HfUploader;

// Upload dataset
let uploader = HfUploader::new("HF_TOKEN")
    .repo_id("username/my-dataset")
    .private(false);

uploader.upload(&dataset)?;

// Specific revision/branch
let uploader = HfUploader::new("HF_TOKEN")
    .repo_id("username/my-dataset")
    .revision("v1.0");
```

## Examples 82-83: Private Upload and Cache

```rust
use alimentar::hf_hub::{HfUploader, HfCache};

// Private repository
let uploader = HfUploader::new("HF_TOKEN")
    .repo_id("username/private-data")
    .private(true);

// Cache management
let cache = HfCache::default();
println!("Cache size: {} bytes", cache.size()?);
cache.clear()?;
```

## Examples 84-85: Offline Mode and Token Auth

```rust
use alimentar::hf_hub::HfDataset;

// Offline mode (use cache only)
let dataset = HfDataset::builder("username/dataset")
    .offline(true)
    .build()?
    .download()?;

// Explicit token authentication
let dataset = HfDataset::builder("username/private-dataset")
    .token("hf_xxxxx")
    .build()?
    .download()?;
```

## CLI Usage

```bash
# Download from Hub
alimentar hf download username/dataset --split train

# Upload to Hub
alimentar hf upload data.parquet username/my-dataset

# With authentication
HF_TOKEN=hf_xxx alimentar hf upload data.parquet username/my-dataset --private

# Cache management
alimentar hf cache --list
alimentar hf cache --clear
```

## Environment Variables

| Variable | Description |
|----------|-------------|
| `HF_TOKEN` | HuggingFace API token |
| `HF_HOME` | Cache directory location |
| `HF_OFFLINE` | Force offline mode (0/1) |

## Key Concepts

- **Dataset cards**: Metadata and documentation
- **Revisions**: Git-like versioning
- **Cache**: Local storage of downloaded datasets
- **Privacy**: Public vs private repositories