sheeter-engine 1.2.0

Convert, inspect, and transform data files across ten formats with streaming where it matters and auto-detection where it can.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126

# Sheeter

Convert, inspect, and transform data files across **ten formats** — with streaming where it matters and auto-detection where it can.

Sheeter is a **conversion-first** tool with **transforms you can chain**. It is not a query language (use `jq`), not a dataframe (use `polars`), not a database client (use `duckdb`). It is what you reach for when you need to move data between formats and apply a quick filter, rename, sort, or dedupe while you are at it.

Formats: `csv`, `tsv`, `json`, `ndjson`, `yaml`, `xml`, `toml`, `xlsx`, `ods`, `sql`.

> **Status: feature-complete.** v0.12.0 is the final minor release before v1.0.0.
> The CLI, `Format` enum, `CanonicalValue` variants, and `convert`/`convert_workbook`
> signatures are frozen. Remaining work before 1.0.0 is bug fixes and docs.

## Install

    cargo install sheeter-engine

The installed binary is `sheeter`.

## Usage

    sheeter convert --from <FMT> --to <FMT> [OPTIONS] <input> <output>

`--from` is optional: the format is detected from the file extension, or from
content when reading stdin. Use `-` for stdin or stdout. ZIP-based formats
(XLSX, ODS) are detected by inspecting the container.

### SQL over files

`sheeter query` accepts a `SELECT` statement and resolves the `FROM` clause
against the filesystem. Any format the engine reads can appear in `FROM`,
so a single query can join an XLSX sheet with an NDJSON stream without
conversion.

    # Group and aggregate across CSV
    sheeter query "SELECT country, SUM(revenue) AS total FROM users.csv GROUP BY country ORDER BY total DESC"

    # Join XLSX with JSON
    sheeter query "SELECT u.name, o.amount FROM users.xlsx u JOIN orders.json o ON u.id = o.user_id"

    # Glob multiple files into one table
    sheeter query 'SELECT * FROM "logs/*.json" ORDER BY ts DESC LIMIT 100'

### The Unix pipeline use case

    # CSV from anywhere, filtered, as NDJSON, into jq
    cat data.csv | sheeter convert --to ndjson --filter "age > 30" - - | jq -c .

    # NDJSON from an API, as a CSV
    curl -s api.example.com/users | sheeter convert --to csv - - | head

    # Auto-detect anything from stdin (including XLSX and ODS)
    some-tool emit | sheeter convert --to json - -

    # What format is this, anyway?
    sheeter detect mystery.bin

### Options

    --sheet NAME      Select one sheet (XLSX/ODS input only)
    --dialect NAME    SQL dialect: generic, postgresql, sqlite, mysql
    --filter EXPR     Filter rows (e.g. "age > 30 and city == 'SP'")
    --select COLS     Keep only these columns (comma-separated)
    --rename PAIRS    Rename columns (e.g. "old=new,foo=bar")
    --limit N         Keep at most N rows
    --sort SPEC       Sort by column(s) (e.g. "age desc,name")
    --dedup           Drop duplicate rows (whole row)
    --dedup-by COLS   Drop duplicates by column(s)

### Examples

    # CSV → ODS, round-trip
    sheeter convert --from csv --to ods data.csv data.ods
    sheeter convert --from ods --to csv data.ods back.csv

    # Multi-sheet XLSX → one CSV per sheet
    sheeter convert --from xlsx --to csv report.xlsx report.csv

    # Sort large CSV in bounded memory
    sheeter convert --from csv --to csv --sort "age desc" big.csv sorted.csv

    # Inspect any data file
    sheeter inspect --format xlsx report.xlsx

    # Display multiple files as formatted tables
    sheeter view data.csv config.tsv report.xlsx

    # Merge files into an XLSX workbook (one sheet per file)
    sheeter xlsx:merge a.csv b.json c.yaml -o combined.xlsx

    # Split every sheet of an XLSX into its own JSON file
    sheeter xlsx:frag combined.xlsx

## How it works

Every reader produces a `CanonicalValue`; every writer consumes one. Adding
a new format costs one reader and one writer, not `N-1` pairwise converters.
`CanonicalValue::Bytes` carries binary columns from Parquet-capable readers
(XLSX, future Parquet), round-tripping them through JSON via `{"$binary": ...}`
and through SQL via `X'hex'`.

## Streaming vs buffered

| Input → Output | Memory | Transforms |
|---|---|---|
| CSV/TSV → CSV/TSV (no `--sort`) | Constant | Per row |
| CSV/TSV → CSV/TSV (`--sort`) | Bounded by 50k-row chunk | With external sort |
| CSV/TSV ↔ NDJSON | Constant | Per row |
| NDJSON ↔ NDJSON | Constant | Per row |
| Any other pair | Full document in memory | Via hub round-trip |

## How this compares

| Tool | Formats | Streaming | Auto-detect (incl. ZIP) | Transforms |
|---|---|---|---|---|
| **Sheeter** | 10 | CSV/TSV/NDJSON | Yes | filter, select, rename, limit, sort, dedup |
| `dasel` | 7 | No | No | Limited |
| `yq` | 3 (YAML/JSON/XML) | No | No | jq syntax |
| `miller` | 5 | CSV/TSV/JSON | No | DSL |
| `qsv` | CSV only | Yes | Schema sniff | Rich |
| `dkit` | 6 | No | Yes | Limited |

## Versioning

This project follows [Semantic Versioning 2.0.0](https://semver.org/spec/v2.0.0.html).
Changelog follows [Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/).
Commit messages follow [Conventional Commits 1.0.0](https://www.conventionalcommits.org/en/v1.0.0/).