sheeter-engine 1.2.0

# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [1.2.0] - 2026-11-02

### Added

- `sheeter view <files...>`: displays one or more data files as formatted
  UTF-8 tables, one per logical table. XLSX and ODS multi-sheet inputs
  produce one table per sheet. `--to-json` emits the hub JSON instead of
  the formatted tables.
- `sheeter xlsx:merge <files...> -o out.xlsx`: appends one sheet per input
  file to an existing or new XLSX workbook. Single-table inputs use the
  filename stem as sheet name; multi-sheet inputs prefix the sheet with the
  file stem (`report_Q1`). Sheet names are truncated to Excel's 31-char
  limit, and collisions get `_2`, `_3` suffixes. Existing sheets are
  preserved.
- `sheeter xlsx:frag <file.xlsx>`: splits every sheet of an XLSX file into
  its own JSON file. Sheet names are NFD-decomposed, accents stripped,
  lowercased, non-alphanumerics collapsed to `_`. `--out DIR` writes
  elsewhere.
- `view::expand`, `view::render`, `xlsx_tools::merge`, `xlsx_tools::frag` in
  the library.
- Dependencies: `comfy-table` for UTF-8 box rendering, `unicode-normalization`
  for accent stripping.

### Notes

- `xlsx:merge` and `xlsx:frag` are Excel-specific by design. The compactness
  of the format makes operations that involve other formats viable in ways
  they are not for ODS.

## [1.1.1] - 2026-11-02

### Fixed

- `sheeter query` now applies SQL NULL comparison semantics: any comparison
  (`=`, `!=`, `<`, `<=`, `>`, `>=`) with a NULL operand evaluates to false,
  so rows with NULL join keys or NULL WHERE operands are excluded. Previously
  NULL was treated as equal to NULL, causing spurious join matches.
- `sheeter query` output to stdout now ends with a newline, so piped commands
  (`| jq .`, `| head`) render cleanly instead of concatenating the prompt.

### Added (tests only)

- Regression tests for xlsx JOIN json, mixed numeric/string join keys, and
  NULL in WHERE clauses.

## [1.1.0] - 2026-11-02

### Added

- `sheeter query "<SQL>" [output]`: SQL SELECT over files, resolved against
  the current directory. The `FROM` clause accepts a single file, a glob
  pattern (`"orders/*.json"`), or a comma-separated list of tables (cross
  join). Every format the engine reads can appear in `FROM`, so a single
  query can join an XLSX sheet with an NDJSON stream without conversion.
- Supported SQL surface: `SELECT` with column references, aliases, and
  arithmetic; `WHERE`; `GROUP BY` with `SUM`, `AVG`, `COUNT`, `MIN`, `MAX`;
  `HAVING` (evaluated on projected aliases); `ORDER BY`; `LIMIT`; `DISTINCT`;
  `INNER JOIN` and `LEFT JOIN` with `ON`.
- Output format is auto-detected from the output filename extension, or
  forced with `--to`. Default for stdout is JSON.
- `query::run(sql, base_dir)` in the library.

### Notes

- Query results are materialized in memory. For files larger than available
  RAM, use `convert --filter` on CSV/NDJSON, which streams.
- Only `SELECT` statements are supported. `INSERT`, `UPDATE`, `DELETE`,
  CTEs, subqueries, and window functions are not.

## [0.12.1] - 2026-11-02

### Changed

- Crate description and `--help` about string aligned with the product scope:
  convert, inspect, and transform. The README now states explicitly what
  Sheeter is (conversion-first with chainable transforms) and is not (a query
  language, a dataframe, a database client).
- `repository` now points to the real GitHub URL.

### Added

- `LICENSE` (MIT) at the repository root.
- `SECURITY.md` with a vulnerability disclosure policy.
- `CONTRIBUTING.md` documenting Conventional Commits, SemVer, and the PR
  validation gate.
- `[package.metadata.docs.rs]` block so docs.rs builds with all features.

## [0.12.0] - 2026-11-02

**Final feature release before API stabilization.**

### Added

- ODS (OpenDocument Spreadsheet) as the tenth format. Reading uses
  `calamine::Ods`; writing emits a minimal, spec-compliant ODS ZIP with
  `mimetype`, `META-INF/manifest.xml`, and `content.xml`.
- `CanonicalValue::Bytes(Vec<u8>)` variant, plus base64 `$binary` marker in
  JSON output and `X'hex'` in SQL output. Binary columns from Parquet and
  XLSX round-trip without corruption.
- ZIP-aware format detection. `detect_content` now recognizes ODS and XLSX
  by inspecting the ZIP container: ODS via its `mimetype` entry, XLSX via
  `[Content_Types].xml`. ODT/ODP are rejected with a clear message.
- `--ods-sheet` is not needed: `--sheet` works for ODS the same way as XLSX.

### Notes

- **The API is now considered feature-complete.** v0.12.0 is the last minor
  release before v1.0.0. Any remaining changes before 1.0.0 are bug fixes and
  documentation; the CLI, `Format` enum, `CanonicalValue` variants, and
  `convert`/`convert_workbook` signatures are frozen pending v1.0.0.

## [0.11.0] - 2026-11-02

### Added

- NDJSON (newline-delimited JSON) as a first-class format. Streaming reader
  and writer for CSV/TSV, NDJSON, and combinations. `Format::Ndjson`,
  `CanonicalValue::from_ndjson`, `CanonicalValue::to_ndjson`,
  `stream_ndjson_to_delimited`, `stream_delimited_to_ndjson`,
  `stream_ndjson_to_ndjson`.
- Content-based format detection via `detect_content` and
  `detect_extension`. Tries JSON, NDJSON, XML, CSV, TSV, TOML, YAML in
  order; first parser that accepts wins.
- `--from` is now optional on `convert`: the format is detected from file
  extension or, for stdin, from content.
- `-` accepted as input (stdin) and output (stdout) on `convert` and
  `inspect`.
- `sheeter detect <file|->` subcommand: prints the detected format.

### Notes

- Binary output (XLSX) cannot go to stdout; the CLI errors explicitly
  instead of writing garbage.
- Streaming works for CSV↔TSV, CSV/TSV↔NDJSON, and NDJSON↔NDJSON. Other
  pairs still buffer through the canonical hub.
- `--sort` with NDJSON input/output is rejected for now; it lands in
  v0.11.1.

## [0.10.0] - 2026-11-02

### Added

- `--sort "col [asc|desc][,col2...]"` on `convert`. For CSV/TSV to CSV/TSV,
  uses external merge sort with temp-file spill: peak memory is bounded by
  the chunk size (50k rows), independent of input size.
- `--dedup` and `--dedup-by "col1,col2"` on `convert`. Removes duplicate
  rows (whole row or by selected columns). Works in streaming, buffered, and
  sorted paths.
- `SortSpec` and `external_sort_delimited` in the library.
- `Transforms::dedup` field.

### Fixed

- `--filter`, `--select`, `--rename`, and `--limit` are no longer silently
  ignored on buffered paths (e.g. `json → yaml`). The buffer path now
  round-trips through the canonical hub, applies transforms, and reserializes.
  Regression tests in `tests/regression_buffer.rs` guard this.
- `stream_delimited` now honors `Transforms::dedup`.

### Notes

- `--sort` on non-CSV/TSV paths is documented as buffered and currently
  applies transforms but not sorting; full buffered sort lands in v0.11.0.
- External sort uses `tempfile::NamedTempFile` under the OS temp directory
  and cleans up on success.

## [0.9.0] - 2026-11-02

### Added

- `inspect` subcommand: prints format, document count, per-document row and
  column counts, and inferred column types.
- `--filter`, `--select`, `--rename`, `--limit` flags on `convert`. Filter
  uses a mini-DSL supporting `==`, `!=`, `<`, `<=`, `>`, `>=`, `and`, and
  literals (numbers, strings, booleans, null).
- `Filter`, `Transforms`, `stream_delimited` in the library.
- CSV and TSV inputs to CSV and TSV outputs now stream: constant memory
  regardless of file size. Transforms apply per-row.

### Notes

- Streaming is limited to CSV/TSV pairs in this release. Other format
  combinations buffer through the canonical hub.
- `--sort` and `--dedup` are not in this release; they require buffering and
  will arrive in v0.9.1.

## [0.8.0] - 2026-11-02

### Added

- SQL support via `sqlparser` 0.62 (pure Rust, ANSI SQL:2011).
- `SqlDialect` enum: `Generic`, `PostgreSql`, `Sqlite`, `MySql`.
- `--dialect` CLI flag for parsing SQL input.
- Multi-table SQL inputs produce one output per table for CSV/TSV, and a
  single merged document for JSON/YAML/XML/TOML, reusing the `Workbook` hub.
- SQL output derives the table name from the output filename stem
  (`users.sql` → `INSERT INTO "users"`).
- `NULL`, booleans, integers, floats, and escaped single quotes are preserved.

### Notes

- Only `INSERT ... VALUES` statements contribute data. `SELECT` without
  `INSERT` errors with a clear message. `CREATE TABLE` is used to recover
  column names when `INSERT` omits the column list.
- SQL output is ANSI-compatible `INSERT` statements; the dialect flag affects
  parsing only.

## [0.7.1] - 2026-11-02

### Fixed

- XLSX reader now coerces whole-valued floats (`1.0`, `42.0`) into JSON
  integers. Excel stores all numeric cells as floats, so a Pokedex `id`
  column previously surfaced as `1.0`, `2.0` instead of `1`, `2`. Fractional
  values (`6.9`, `13.5`) are untouched.

## [0.7.0] - 2026-11-02

### Added

- XLSX support via `calamine` (reader) and `rust_xlsxwriter` (writer), both
  pure Rust with no C dependencies.
- `Format::Xlsx`, `CanonicalValue::from_xlsx_workbook`,
  `CanonicalValue::to_xlsx_workbook`.
- `convert_workbook(from, to, bytes)` returns one `(sheet_name, content)`
  pair per output document, enabling multi-sheet workbooks.
- CLI `--sheet NAME` flag to extract a single sheet by name.
- Multi-sheet XLSX to CSV/TSV produces one file per sheet, using the pattern
  `<stem>.<SheetName>.<ext>`. Collisions after sanitization get `_2`, `_3`
  suffixes.

### Changed

- `convert(from, to, &str)` now rejects `Format::Xlsx` with a pointer to
  `convert_workbook`; XLSX is binary and cannot round-trip through `&str`.
- JSON/YAML/XML/TOML outputs from multi-sheet XLSX emit a single document
  whose root is an object keyed by sheet name.
- `Format::ALL` now has 7 entries.

### Notes

- Dates are stringified (ISO-8601), matching TOML and YAML behavior.
- Formulas are read as their cached computed values, if present. The Sheeter
  never evaluates formulas.
- Formatting, merged cells, charts, images, and pivot tables are ignored.
- Writing is `.xlsx` only. Reading `.xls` and `.ods` via `calamine` is not
  wired to the CLI in this release.

## [0.6.0] - 2026-11-02

### Changed

- CLI collapsed from 30 pairwise subcommands to a single `convert`
  subcommand with `--from` and `--to` flags. Usage becomes
  `sheeter convert --from <FMT> --to <FMT> <input> <output>`.
- New `Format` enum and `convert(from, to, input)` function in the library
  expose the hub directly; the 30 pairwise functions remain available.

### Added

- Property-based tests with `proptest` for `csv → json → csv` and
  `tsv → json → tsv` round-trip byte equality.

### BREAKING CHANGE

- CLI: `sheeter csv-to-json` and the other 29 pairwise subcommands were
  replaced by `sheeter convert --from csv --to json`. Scripts and CI
  invoking the old syntax must be updated.

## [0.5.0] - 2026-11-02

### Added

- TOML support via the `toml` crate (pure Rust): `toml_to_json`,
  `json_to_toml`.
- Cross-format TOML wrappers: `toml_to_csv`, `toml_to_tsv`, `toml_to_yaml`,
  `toml_to_xml`, `csv_to_toml`, `tsv_to_toml`, `yaml_to_toml`, `xml_to_toml`.
- CLI subcommands for all 10 new routes.
- TOML tabular data is discovered via the first array-of-tables found at any
  depth, so nested configurations produce CSV/TSV without manual unwrapping.

### Changed

- `as_tabular` now accepts flat objects (all scalar values) as a single row,
  enabling `toml_to_csv` on simple documents and making `<r>just text</r>`
  produce `r\njust text\n` instead of erroring.

### Notes

- TOML datetimes are stringified, matching YAML behavior. Round-trip back to
  TOML restores them as strings, not as date types.
- `null` values in JSON are skipped during JSON→TOML conversion (TOML has no
  null type). JSON arrays containing null error out.
- JSON arrays at root are wrapped in `{item: [...]}` on TOML output, matching
  the XML wrapper convention.

### BREAKING CHANGE

- `as_tabular` behavior: XML with a scalar root (`<r>just text</r>`) now
  produces a single-row CSV output instead of erroring. Consumers that
  relied on the previous error should check for this input shape explicitly.

## [0.4.0] - 2026-11-02

### Added

- `CanonicalValue` type: format-agnostic intermediate representation that
  replaces the JSON string hub. Every reader produces a `CanonicalValue`;
  every writer consumes one.
- XML support via `quick-xml` (pure Rust, streaming, no C dependencies):
  `xml_to_json`, `json_to_xml`.
- Cross-format XML wrappers: `csv_to_xml`, `tsv_to_xml`, `yaml_to_xml`,
  `xml_to_csv`, `xml_to_tsv`, `xml_to_yaml`.
- CLI subcommands for every new conversion route.

### Changed

- The hub is no longer a serialized JSON string. Conversions between formats
  that are not JSON compose through `CanonicalValue`, so adding a new format
  now costs one reader and one writer, not `N-1` pairwise converters.

### Notes

- XML documents are represented as a single-key object whose key is the root
  element name. Attributes use `@name` keys; text uses `#text`. Repeated
  sibling elements become arrays. `xml_to_csv` and `xml_to_tsv` extract the
  tabular array when the XML follows `<root><item>...</item></root>`.

## [0.3.1] - 2026-11-02

### Fixed

- `yaml_to_json` now treats `!!binary`-tagged scalars as literal strings instead
  of silently base64-decoding them, preventing value corruption for annotated
  YAML documents.

### Added

- Regression tests for ragged-row rejection in CSV/TSV and `!!binary`
  string preservation.
- Documented that non-string YAML keys are stringified, matching mainstream
  YAML-to-JSON converters.

## [0.3.0] - 2026-11-02

### Added

- YAML output for all tabular sources: `csv_to_yaml`, `tsv_to_yaml`, `json_to_yaml`.
- YAML input for all tabular targets: `yaml_to_csv`, `yaml_to_tsv`, `yaml_to_json`.
- Direct `csv_to_tsv` and `tsv_to_csv` conversions.
- CLI subcommands for every new route.
- YAML support uses `serde-saphyr` (pure Rust, YAML 1.2, no C dependency).

### Fixed

- `csv_to_json` / `tsv_to_json` now strip UTF-8 BOM before parsing, so
  Excel-exported files no longer produce headers like `\u{feff}name`.
- `yaml_to_json` uses `strict_booleans` to avoid the YAML 1.1 "Norway problem":
  `yes`/`no`/`on`/`off` are kept as strings, matching YAML 1.2 semantics.
- `json_to_yaml` and `yaml_to_json` preserve `null` values instead of
  stringifying them.
- `json_to_csv` / `json_to_tsv` build the header from the union of all object
  keys in first-appearance order, guaranteeing stable round-trips regardless
  of per-object key ordering.
- `json_to_tsv` correctly quotes cells containing tab characters, preventing
  line-splitting corruption.
- Dependencies pinned to `serde-saphyr` `0.0.29` (pure Rust, YAML 1.2, no C
  dependency) to keep the crate portable to Android/Termux.

## [0.2.0] - 2026-11-02

### Added

- `tsv_to_json` and `json_to_tsv` library functions.
- `tsv-to-json` and `json-to-tsv` CLI subcommands.
- TSV support reuses the same lossless conversion engine as CSV, with
  identical duplicate-header and nested-value rejection rules.

## [0.1.2] - 2026-11-02

### Fixed

- `csv_to_json` now errors on duplicate CSV headers instead of silently
  dropping columns.
- `json_to_csv` now collects the union of keys across all objects instead
  of using only the first object's keys, which silently dropped data.
- `json_to_csv` now errors on nested JSON objects/arrays instead of
  stringifying them and corrupting round-trips.

## [0.1.1] - 2026-11-02

### Changed

- Rename crate from `sheeter` to `sheeter-engine` to avoid collision with an
  existing crate on crates.io. Import path changes from `sheeter::` to
  `sheeter_engine::`.

## [0.1.0] - 2026-11-02

### Added

- Initial release of Sheeter CLI.
- CSV to JSON conversion.
- JSON to CSV conversion.
- Support for reading from and writing to files.