reshard-tokenized 0.1.0

CLI for merging tokenized shard .npy files and remapping .csv.gz metadata offsets.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

<p align="center">
  <img src="https://github.com/soldni/reshard-tokenized/blob/master/assets/logo.png?raw=true" alt="Library logo" width="384"/>
</p>

# reshard-tokenized

A Rust CLI that merges tokenized shard files from a directory tree into one or more output shards:
- `.npy` files are concatenated per shard
- `.csv.gz` metadata rows are merged with remapped offsets

## Prerequisites

- Rust toolchain (stable)
- Cargo (included with Rust)

## Build

```bash
cargo build
```

## Run

```bash
cargo run -- \
  --input-path tests/data/tokenized_input \
  --num-files 2 \
  --output-path /tmp/merged
```

CLI help:

```bash
cargo run -- --help
```

## Test

```bash
cargo test
```

Integration tests use fixtures in `tests/data/tokenized_input`.

## Output

- If `--num-files 1`, output is written as `<output-path>.npy` and `<output-path>.csv.gz`
- If `--num-files > 1`, output is written as:
  - `<output-path>/00000000.npy`, `<output-path>/00000000.csv.gz`
  - `<output-path>/00000001.npy`, `<output-path>/00000001.csv.gz`
  - ...