alimentar 0.2.7

Data Loading, Distribution and Tooling in Pure Rust
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142

# Federated & Splitting (Examples 66-75)

This section covers dataset splitting for ML and federated learning.

## Examples 66-67: Train/Test and Stratified Split

```rust
use alimentar::{ArrowDataset, DatasetSplit};

let dataset = ArrowDataset::from_parquet("data.parquet")?;

// Basic 80/20 split
let split = DatasetSplit::from_ratios(
    &dataset,
    0.8,      // train
    0.2,      // test
    None,     // no validation
    Some(42)  // seed for reproducibility
)?;

// Stratified by label column
let split = DatasetSplit::stratified(
    &dataset,
    "label",  // stratify column
    0.8, 0.2, None,
    Some(42)
)?;

assert_eq!(split.train().len() + split.test().len(), dataset.len());
```

## Examples 68-69: K-Fold and Leave-One-Out

```rust
use alimentar::DatasetSplit;

// 5-fold cross-validation
let folds = DatasetSplit::kfold(&dataset, 5, Some(42))?;
for (i, (train, test)) in folds.iter().enumerate() {
    println!("Fold {}: train={}, test={}", i, train.len(), test.len());
}

// Leave-one-out
let loo = DatasetSplit::leave_one_out(&dataset)?;
```

## Examples 70-71: Node Manifest and Coordinator

```rust
use alimentar::{DatasetSplit, NodeSplitManifest, FederatedCoordinator};

let split = DatasetSplit::from_ratios(&dataset, 0.8, 0.2, None, Some(42))?;
let manifest = NodeSplitManifest::from_split("node1", &split);

println!("Node: {}", manifest.node_id);
println!("Train rows: {}", manifest.train_rows);
println!("Test rows: {}", manifest.test_rows);

// Coordinator aggregates manifests
let coordinator = FederatedCoordinator::new();
coordinator.register_node(manifest)?;
```

## Examples 72-74: IID/Non-IID/Dirichlet Strategies

```rust
use alimentar::{FederatedSplit, PartitionStrategy};

// IID (random) partitioning
let splits = FederatedSplit::partition(
    &dataset,
    10, // 10 nodes
    PartitionStrategy::IID,
    Some(42)
)?;

// Non-IID (label-skewed)
let splits = FederatedSplit::partition(
    &dataset,
    10,
    PartitionStrategy::NonIID { skew: 0.5 },
    Some(42)
)?;

// Dirichlet distribution
let splits = FederatedSplit::partition(
    &dataset,
    10,
    PartitionStrategy::Dirichlet { alpha: 0.5 },
    Some(42)
)?;
```

## Example 75: Multi-Node Simulation

```rust
use alimentar::{FederatedSplit, FederatedCoordinator};

let coordinator = FederatedCoordinator::new();

// Distribute to 10 simulated nodes
let splits = FederatedSplit::partition(&dataset, 10,
    PartitionStrategy::IID, Some(42))?;

for (i, split) in splits.iter().enumerate() {
    let manifest = NodeSplitManifest::from_split(
        &format!("node_{}", i),
        split
    );
    coordinator.register_node(manifest)?;
}

// Verify distribution
let stats = coordinator.distribution_stats()?;
println!("Total: {} rows across {} nodes", stats.total_rows, stats.node_count);
```

## CLI Usage

```bash
# Basic split
alimentar fed split data.parquet --train 0.8 --test 0.2

# Stratified split
alimentar fed split data.parquet --stratify label --train 0.8 --test 0.2

# Create node manifest
alimentar fed manifest data.parquet --node-id node1

# Plan federated distribution
alimentar fed plan --nodes 10 --strategy iid data.parquet

# Verify manifests
alimentar fed verify manifest1.json manifest2.json
```

## Key Concepts

- **Reproducibility**: Seed ensures same split
- **Stratification**: Preserves class distribution
- **Manifest**: Metadata about node's data
- **Coordinator**: Central aggregation point