# Changelog
All notable changes to `hayson-engine` are documented here. Format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
## 0.27.0 — 2026-10-16
### Added
- `database::convert_encryption(dir, &ConvertOptions { from, to, lock })` seals a plain database,
unseals an encrypted one or re-seals it under other keys, in place and recoverably (stage, verify with
the new keys, swap through `.restore-old`, roll back on failure). Log lines are re-sealed with fresh
IVs, image files are renamed (plain content hash ↔ keyed hash) and the records that name them are
rewritten, index snapshots are dropped. Returns `ConvertResult { files, tables, images, lines_renamed }`.
### Changed
- The restore machinery lets its staging step set the files to install and replace a database whose
keys differ from the ones it is replaced with (`RestorePart::current_encryption`).
## 0.26.0 — 2026-10-16
### Added
- `OpenOptions::encryption` (`CipherOptions`): a database opened with keys seals everything it
writes with AES-256-GCM — the metadata document, every log line (`$hayson-seal$v1$<keyId>$<iv>$<ct+tag>`,
one text line, so offsets and the half-written-tail rule are unchanged), the index snapshots, the
unit-of-work journal and the image files (named by a BLAKE2b-MAC of their bytes under a key derived
from the database key). Registries and indexes are rebuilt in memory from the decrypted lines.
A wrong key, a missing key and a key for a plain database are refused as `DatabaseError::Cipher`;
a line that does not authenticate is a `HaysonCorruptLogError` with its offset.
`crypto::Sealer` is the sealing primitive; `storage::LineCodec` lets a log store its lines through one.
- `encryption` on `BackupOptions`, `RestoreOptions`, `PackOptions` (and `PackInclude`, per database),
`UnpackOptions` (plus `include_encryption`) and `VerifyOptions`: the files travel still sealed, the
keys are used to read which files a database owns and to verify. `verify` without keys reports
`catalog-encrypted`; with wrong keys `catalog-key-rejected`; with keys for a plain database
`catalog-not-encrypted`.
- `CipherOptions` is `PartialEq`/`Eq`.
### Fixed
- An archive entry whose bytes fail the checksum is now a refusal of the archive
(`DatabaseError::Schema`, "is damaged") instead of a plain io error.
### Changed
- Compaction of an encrypted log re-seals each line, so it needs the key.
- `migrate_layout` refuses an encrypted database.
- A restore or unpack that refuses removes the directories it made, including those of nested databases.
- Depends directly on `blake2` 0.11 (already in the build through `argon2`).
## 0.25.0 — 2026-10-15
### Added
- `PackOptions::include` (`PackInclude { path, exclude_tables }`): databases in subfolders of the
packed directory travel in the same archive under their folder, each with a
`<folder>/hayson-backup.json`; the main manifest lists the folders in `databases`.
`PackResult::databases` reports files and bytes per database (`PackedDatabase`).
- `unpack_database` installs all databases of such an archive together: every one is staged and
verified before any is swapped in, and the swap runs through a single `.restore-old`, so one
failure puts all of them back. An included path that repeats, sits inside another included one,
or is not a database is refused when packing.
### Changed
- The restore machinery works on a list of databases (`restore_database` is the one-database case).
## 0.24.0 — 2026-10-15
### Added
- `BackupOptions::exclude_tables` and `PackOptions::exclude_tables`: the data of the named tables
(logs, partitions, images, snapshots) stays out of the backup or archive and its manifest. Their
declarations stay in the metadata document and an unpartitioned table's log is written empty, so
the copy opens read-only with those tables declared and empty. An undeclared name is refused.
- `UnpackOptions::max_image_bytes` and `RestoreOptions::max_image_bytes`: an image over the limit
refuses the unpack/restore before anything is replaced, and a target directory the call created
is removed again after any refusal.
### Changed
- `unpack_database` refuses archives the zip reader tolerates: bytes after the end-of-central-
directory record, entries whose local header names them differently from the directory, names
that are not UTF-8, and two entries sharing a name (the reader silently collapsed them).
- `verify` no longer reports the logs of a nested database (a subfolder with its own metadata
document) as `log-undeclared`.
## 0.23.0 — 2026-10-15
### Added
- `passwordHash: true` on a `string` field: the value is stored as a salted Argon2id hash,
`$hayson-pw$v1$argon2id$m=…,t=…,p=…$<salt>$<hash>`, and never as text. `Table::verify_password`
answers `PasswordCheck { ok, needs_rehash }` in constant time and with the same amount of work
for an unknown record as for a wrong password. `OpenOptions::password_hash` sets the cost of new
hashes (`PasswordHashParams`, default 19 MiB / 2 passes / 1 lane); a stored hash keeps its own.
The field may be combined with `cypher`; it rejects `index`, `unique`, `fulltext` and `default`.
The answer of a staged write leaves out a password that is not yet hashed.
- Depends on the `argon2` crate (RustCrypto, MIT OR Apache-2.0, default features off): new in the
build are `argon2`, `base64ct`, `blake2` and `digest`.
## 0.22.0 — 2026-10-15
### Added
- `OpenOptions::max_image_bytes` and `Table::set_max_image_bytes`: a write whose image field
decodes to more bytes than the limit is refused before anything is staged, with the typed
`errors::HaysonImageTooLargeError { bytes, max_bytes }` boxed in the `io::Error`.
## 0.21.0 — 2026-10-15
### Added
- `errors::HaysonImageClashError { physical_path }`: a write of an image whose bytes differ from the
file stored under the same content hash is refused with it (boxed in the `io::Error`, same
message as before) so a binding can tell it from a disk failure.
### Changed
- A corrupt log line met while scanning or reading a log now travels as the typed
`HaysonCorruptLogError` boxed in the `io::Error` (kind `InvalidData`, same text) instead of being
flattened to a string, so its `path` and `offset` survive.
## 0.20.0 — 2026-10-15
### Added
- `database::pack_database(dir, dest, &PackOptions)` writes the files a database accounts for (the
same set as `backup_database`) into one zip archive at `dest`, stored without compression, with a
`hayson-backup.json` manifest and any `extra` entries the caller supplies. It holds the writer
lock, builds the archive beside `dest` and renames it into place, and enforces the size limits
while writing so a pack can always be read back.
- `database::unpack_database(archive, dir, &UnpackOptions)` installs such an archive with the
recoverable procedure of `restore_database`. The archive is judged before anything is written:
`PackLimits` (archive 256 MiB, 10 000 entries, 128 MiB per entry, 256 MiB in total by default),
entry names (no `..`, absolute or drive paths, backslashes, control characters, device names, or
names equal ignoring case), symbolic links, compressed or encrypted entries, and declared sizes
against the manifest and against what each entry really holds.
- `database::read_pack_entry(archive, name, &PackLimits)` reads one entry of a pack after the same
checks.
- Depends on the `zip` crate (MIT, default features off, so no compression code is built).
## 0.19.0 — 2026-10-14
### Added
- `database::backup_database(dir, dest, &BackupOptions)` copies the files a database accounts for —
its metadata document, table logs, image directories and, with `indexes`, the `.idx` snapshots —
into `dest` with a `hayson-backup.json` manifest. It holds the writer lock for the copy, builds
the backup beside `dest` and renames it into place, and leaves out anything the database does not
own (a database in a subfolder, the lock file). A database with an unfinished unit of work is
refused.
- `database::restore_database(backup, dir, &RestoreOptions)` restores such a backup recoverably: it
checks the backup against its manifest and `verify`, moves the files being replaced into
`.restore-old`, moves the new ones in and only then deletes `.restore-old`. A failed move puts the
previous files back (or reports both errors); a `.restore-old` left by an interrupted restore is
refused.
## 0.18.0 — 2026-10-13
### Added
- Image blob API on `Table`: `read_image(physical_path)`, `list_images()` (file size and the live
records referencing it; an empty `referenced_by` is an orphan, a reference to a file that is gone
reports `is_missing()`), `image_usage()` (count, bytes, orphan count and bytes) and
`delete_image(physical_path)` (refuses while a live record still references the file).
- `verify` checks image files (`VerifyOptions::images`, default on): `image-missing`,
`image-corrupt` (bytes no longer hash to the file name), `image-orphan` and, with
`max_image_bytes`, `image-oversize`.
### Fixed
- A write whose image bytes differ from the file already stored under the same content hash is
refused before anything is staged. FNV-1a 64 is not collision-resistant, so a colliding image used
to resolve silently to the earlier image's bytes.
## 0.17.0 — 2026-10-12
### Added
- `database::verify(dir, &VerifyOptions)` returns a `VerifyReport` of `Problem`s (stable `code`,
`Severity::Error`/`Warning`, table, file, byte offset) for a database directory. It takes no
lock, writes nothing, and reports damage instead of failing on it: a missing or malformed
metadata document, an unreadable or corrupt log line, an unfinished tail, a log the catalog does
not declare, an unfinished unit-of-work journal and (optionally) relation orphans.
## 0.16.0 — 2026-09-29
### Fixed
- `Database::close()` releases the writer lock even when a table refuses its flush or snapshot
(a log swapped out under the handle, a full disk). The lock file used to outlive the handle,
naming a live pid no staleness rule reclaims, so the next open in the same process waited the
whole lease timeout out. Every table still gets its turn; the first refusal is what the caller
gets back.
- `Table::snapshot()` and `Table::compact()` — including the snapshot `close()` writes and the
compaction the dead-bytes policy triggers — now refuse with `HaysonLogReplacedError` when the
log file at the path is no longer the one the handle holds open, per partition. A snapshot
used to describe the orphaned inode (so the next open failed with
`HaysonIndexMismatchError`), and a compaction used to overwrite the file a checkout had put at
the path with the orphaned inode's contents.
- A refused append no longer leaves a ghost in the staging window: the write's pending claims are
released and the record is dropped from the table's staged reads. Before, a read kept answering
with a record nowhere on disk, a retry of the same key failed with
`HaysonPendingConflictError` instead of the real refusal, and the next flush fired the triggers
of a write that never happened. A batch now leaves the queue only once it landed; a refusal
abandons it and every batch queued behind it, while batches that already landed stay committed.
## 0.15.0 — 2026-09-29
### Fixed
- `Database::open` re-reads the on-disk arrangement (flat/nested) under the writer lock and
moves the lock to where that arrangement keeps it. Before, the arrangement was probed only
once, before the lock: a `migrate_layout` that landed between the probe and the lock had moved
every file and released the old arrangement's lock, so the writer took a lock nobody contends
for and opened empty logs beside the real ones under `tables/` — and the next move did not see
it. A stale nested probe no longer leaves an empty `tables/`, `index/`, `runtime/` in a flat
database either: only the lock's own directory is created before the lock is held.
## 0.14.0 — 2026-09-29
### Added — `ORDER BY`, `LIMIT` and `OFFSET` in the text query language
`parse_query` now accepts, after the WHERE fragment and in this order, `ORDER BY <field> [ASC|DESC]`
(one field, `ASC` by default — `OrderBy` holds a single key), `LIMIT <n>` and `OFFSET <n>`, each at
most once. They map onto the `QuerySpec` fields `order_by`, `limit` and `offset` exactly as
`QueryBuilder::order_by`/`limit`/`offset` do, so `parse_query("age > 30 ORDER BY age DESC LIMIT 10
OFFSET 20")` equals the builder chain. Keywords are case-insensitive like `AND`/`OR`; the field
name follows the identifier rule of predicates. The WHERE fragment may be empty when at least one
clause follows (`ORDER BY name LIMIT 5`); an empty text is still rejected as before. A negative or
fractional count, a clause given twice, or a clause out of SQL order (`LIMIT` before `ORDER BY`)
fails with `HaysonQueryParseError` naming the clause and the position. A field named `limit`,
`offset` or `order` keeps working in a predicate — the keyword is only recognised where a clause
can start.
## 0.13.0 — 2026-09-28
### Added — a write onto a log file swapped out under the engine is refused, not lost
`LogFile::append` now checks, before every append, that the path still leads to the file the
handle holds open (inode where the host reports one, size otherwise — `LogFile::replaced_by`,
which `Table::refresh` shares). A file unlinked or replaced from outside — a `git checkout` of a
data directory kept under version control while a long-running process still holds it open, a
restore from backup — used to receive the append in its orphaned inode, where it vanished with the
inode while every read from memory kept showing the record. Such an append now fails with an
`io::Error` boxing the new `HaysonLogReplacedError { path }`; the database has to be reopened to
attach to the file now at that path. On `wasm32-wasip1` the inode now comes from the host's
`filestat` (real under `node:wasi`), so a same-size replacement is caught there too.
## 0.12.0 — 2026-09-22
### Added — `onDelete` is checked at `delete`, not only diagnosed on request
`Database::delete_checked` — the write path that honours a declared `onDelete`: `restrict` refuses
a `delete` while a live record still references the key (`HaysonRelationRestrictError`, new
`TableError::RelationRestrict`), `cascade` deletes every referencing record too (recursively,
cycle-safe), `set-null` clears the referencing field instead, and `no-action`/undeclared leaves
referencing records untouched exactly as `Table::delete` always has. Everything a cascade needs
lands in one `UnitOfWork`, so a `restrict` found several tables deep aborts the whole delete with
nothing written. A table nothing points at skips the plan and goes straight through `Table::delete`
— the common case pays nothing extra.
`Table::keys_referencing(field, value)` — every live key holding `value` on `field`, via the
registry's `exact`/`sorted` index when one covers the field, a full scan otherwise (the same cost
`check_integrity` already pays for the same question — no reverse index is guaranteed to exist on
a relation's `from.field`).
## 0.11.0 — 2026-09-22
### Added — `catalog::is_format_one_directory`
Whether `migrate_format1(dir)` would find something to rewrite right now — the same guard clauses
it runs before touching disk, without the write. `false` on a directory already at format 2, or on
one with neither a catalog nor any `<table>.schema.json` (a fresh, empty database directory).
Raised by a consumer of `@theritual/hayson` (`docs/consumer-feedback-cerebro.md`): `migrateFormat`
refuses an empty directory, and nothing exported let a caller check for that up front without
duplicating the engine's own `schema_files_of` logic by hand (`readdirSync(dir).some(f =>
f.endsWith('.schema.json'))`) — this is that check, reusing the real one.
## 0.10.0 — 2026-09-22
### Added — a declared relation's `enforce: true` is checked at write, not only diagnosed on request
`catalog::declare_relation` — adds a named relation to the catalog, mirroring `declare_index`.
With `enforce: true`, refuses whole if a live record already breaks it (the same orphan check
`check_integrity` runs, scoped to the one relation) — unless `force: true`, which declares it
anyway and lets enforcement start from here on, existing orphans left as they are.
`Database::put_checked`/`patch_checked` — the write path that actually honours a declared
`enforce: true`: refuses a `put`/`patch` whose `from.field` value has no matching record on the
relation's `to` side, before touching disk. `Table::put`/`patch` are unchanged and still do not
check relations on their own — `Table` cannot see another table's registry, only `Database` holds
every table at once, so a caller that wants enforcement has to come in through `Database`.
`Database::stage_put_checked`/`stage_patch_checked` — the same check for the staged write path
(`Table::stage_put`/`stage_patch`) that `hayson-napi` and `hayson-wasm` stage before flushing.
Both run synchronously with full `Database` access at the call site, same as a direct `put`, so no
separate design was needed for the queued path.
`Table::contains_value(field, value)` — new public method, O(1) via the registry (the primary-key
case is `has()`, the unique-field case is the same lookup `assert_unique` runs for the opposite
question). `HaysonRelationViolationError` / `TableError::Relation` — the new refusal shape.
Deliberately not covered by this release: the async write queue `hayson-napi`/`hayson-wasm` stage
writes through (`stage_put`/`stage_patch`, ADR-mp5p) and multi-table `UnitOfWork` commits — both
still bypass enforcement. See ADR-m4a7 (cerebro) for the full scope and what is left open.
## 0.9.0 — 2026-09-18
### Added — `catalog::migrate_format1`, shared by `hayson-cli` and both bindings
The format-1-to-2 directory rewrite (a `<table>.schema.json` per table, next to the logs, into one
metadata catalog) moved here from `hayson-cli`, so `hayson-napi` and `hayson-wasm` can expose it as
`migrateFormat1()` too, instead of it staying a CLI-only escape hatch. `hayson-cli`'s
`migrate-format1` command now delegates to `catalog::migrate_format1`; its behavior is unchanged.
## 0.8.0 — 2026-09-11
### Changed — a `cypher` field is now always sealed on write
A value written into a field declared `cypher` is encrypted whatever it looks like. Until now the
engine skipped a value whose string started with `$cypher$`, taking the shape for proof that it was
already an envelope — so a caller who wrote such a string, by accident or on purpose, got it stored
verbatim, in the clear, in a field the schema promises is encrypted.
**If you relied on the old behaviour** to move an already-encrypted value between databases, that
value is now sealed a second time instead of being carried over. Use `Table::put_sealed(document,
sealed_fields, force)` instead, which stores the fields it names exactly as given. Nothing else
changes: the envelope format and the AAD are untouched, and a database written by 0.7.0 is read and
written the same as before.
### Added
- `Table::put_sealed` — a write that carries the `cypher` fields it names over as they stand, for a
backup being restored or a copy moving ciphertext between databases. The claim is checked as far
as the keyring allows: the value has to be a well-formed `v1` envelope, and when the key it names
is on hand it also has to authenticate against that very table, record and field — an envelope
copied into the wrong record is refused at the write rather than found unreadable later. A field
whose key is not on hand is taken on the caller's word, which is what lets a restore run without
holding any key. It is deliberately not reachable from a document and is wired to no binding: an
in-process Rust API for trusted callers only.
## 0.7.0 — 2026-09-11
- `database::migrate_layout` — moves a database directory between the `flat` and `nested` on-disk
arrangements, with `LayoutMigrationOptions`, `LayoutMigrationResult` and `LayoutMove` alongside
it. The port of `packages/hayson/src/database/migrate-layout.ts`, which was the last filesystem
implementation left in the TypeScript package and the only writer of a database directory that
did not take this engine's lock: it used the TypeScript lock protocol, deciding ownership by
`pid` where `lock::DirLock` decides it by a minted `owner` token, so a move and an engine writer
could not see each other's lock file. The move now takes `DirLock` on the very path
`Database::open` takes it on, which makes the two mutually exclusive by construction.
- Both on-disk formats move: a format-2 catalog is rewritten with its `layout` named out loud, and
a format-1 manifest keeps its `<table>.schema.json` files and stays on format 1, with `layout`
added or dropped and every member this build does not know carried through untouched.
## 0.6.3 — 2026-09-10
- Whitespace only: `rustfmt` applied to `src/crypto/mod.rs` (0.6.2's edit had drifted from repo
formatting). No behaviour or API change.
## 0.6.2 — 2026-09-10
- Restores five names to the curated public surface that 0.6.0 (the curation pass) missed: the
consumer check that pass ran only covered `crates/`, not `apps/hayson-browser/src-tauri`, which
reaches `hayson-engine` directly. `crypto::{cypher_fields_of, is_envelope}`,
`schema::assert_partition_value`, `catalog::sugar_index_name`, and
`table::{partition_names_in, LOG_SUFFIX}` (re-exported from the now-private `storage`, alongside
the already-public `table::LogOp`) are public again. No behaviour change.
## 0.6.1 — 2026-09-10
- `DirLock::inspect` and `LockState` — read what a `.hayson.lock` says (`Free`, `Held`,
`LeaseExpired`, `HolderGone`, `Unreadable`) without claiming, reclaiming, renewing or removing
anything, so an observer can report on a directory another writer holds.
`LockState::blocks_a_writer` answers whether a writer opening the directory at that moment would
have to wait, which is not the same as "a lock file is there": a lock whose holder is proven dead
is seized with no wait at all.
- `Database::resolve_layout` is now public — where a directory keeps its lock file, metadata
document, logs and snapshots, worked out without opening the database, taking the lock or
creating anything.
## 0.6.0 — 2026-09-10
- **Breaking:** the crate's public surface is now a hand-curated list. `checkpoint`, `queue`,
`registry` and `storage` are private modules — the log files, the write queue, the in-memory
registries and the index snapshots are internal mechanics, not API — and every submodule of a
public module (`catalog::types`, `query::planner`, `schema::validate`, `crypto::field_cipher`
and the rest) is `pub(crate)`. Import the re-exported name instead of the module path:
`query::QuerySpec` for `query::planner::QuerySpec`, `query::ComparisonOp` for
`query::predicates::ComparisonOp`, `query::SearchMode` for `registry::fulltext_index::SearchMode`.
- **Breaking:** names that were only ever crate-internal plumbing left the surface —
`compaction::compact_log`, `catalog::{index_identity, parse_index_entry, parse_trigger_entry,
relations_touching, serialize_catalog}`, `crypto::{cypher_fields_of, envelope_key_id, is_envelope,
open_record, seal_record}`, `schema::{assert_partition_value, broken_links, decode_image_content,
extract_links, is_physical_path, physical_path_of, project_to_schema, undeclared_fields,
without_undeclared}`, `table::{describe_table_document, effect_document, matching, TriggerFiring,
DEFAULT_LARGE_BYTES}`, `txn::{apply_prepared, recover_incomplete, StagedIndex, StagedOverlay}`
and `CompactionResult::slots`.
- `table::PutOutcome` is re-exported at last: `Table::put_with_warnings` and
`Table::patch_with_warnings` return it, but no public path could name it.
## 0.5.0 — 2026-09-10
- `markdown` fields gain `addressable` (default `true`): set `false` and the field holds content
whose links are still validated against the virtual-path index, but the field itself carries no
virtual path of its own — nothing can link to it, and it needs no `virtualPath` value.
- `Table::broken_links_of` and `PutOutcome::warnings` now return `Vec<BrokenLink>` (`{field, path}`)
instead of a formatted sentence per broken link — **breaking** for any Rust caller matching on
the old `Vec<String>` shape.
- `Table::rotate_cypher_keys` — eager key rotation: walks every record, re-encrypts any `cypher`
field not already on the table's current write key, leaves already-current records untouched
(idempotent), and is a fast no-op on a table with no `cypher` fields.
## 0.4.0 — 2026-09-10
- **Breaking:** the schema document's version marker is now `"schemaVersion"`, not `"hayson"` —
`schema::parse` rejects the old key outright. A schema document is caller-authored input, not a
file on disk, so there is nothing to migrate; callers update the literal they pass to
`createTable`/`declareIndex`. See ADR-68rg (cerebro).
- The catalog manifest (`hayson.json`) marker is now `"catalogVersion"` — reading still accepts the
historical `"hayson"` key as a fallback, but every write now emits only the new name.
- The index snapshot (`<table>.idx`) header marker is now `"snapshotVersion"` — an old-keyed
snapshot simply fails to parse and falls back to the existing rebuild-from-log path, unchanged.
- The unit-of-work journal (`.hayson.txn`) header marker is now `"journalVersion"` — reading
accepts either key (an interrupted commit from just before an upgrade is still recovered
correctly), writing emits only the new name.
## 0.3.1 — 2026-09-09
- `open_temporary`'s exclusive-create retry now also treats WASI's `ENOTCAPABLE` (raised when the
create path resolves through a symlink escaping the sandbox) as the same "something's already
there" case native's `EEXIST` already covers, instead of surfacing it as an unhandled IO error.
## 0.3.0 — 2026-09-09
- `Table` gains an optional bounded LRU cache of parsed documents (`OpenOptions::cache`,
`CachePolicy`), off by default. Small and large documents (by approximate size, not a schema
marker) get independent capacity tiers, so a handful of big records can't evict every small one.
Invalidated on every write, unit-of-work commit, journal recovery, foreign-write catch-up
(`refresh`), and compaction — a cache hit skips the log read and JSON parse, nothing else about
`Table::get`'s behavior changes.
## 0.2.0 — 2026-09-09
- `export`'s `physicalPath` acceptance is validated against the engine's own content-hash filename
shape (`is_physical_path`); a caller-supplied path naming an arbitrary file is now refused by
schema validation instead of accepted for a consumer to copy.
- `migrate_format_1_to_2`'s pre-migration `unique` check now resolves logs for partitioned and
`nested`-layout databases, not just flat ones — it used to silently report "no violations" for
either shape, which could let migration write a catalog asserting uniqueness the data already
violated.
## 0.1.0 — initial release
- Storage engine for hayson document databases: append-only JSONL tables (optionally partitioned
by a field's value), a `hayson.json` catalog of table schemas, and in-memory registries for
key lookups, field indexes and full-text search.
- Query engine (`where`/`orderBy`/`limit`, unique and partial indexes), write queue with batching
and `force` override, checkpoint/compaction, cross-table units of work, multi-process access
through a directory lock, and at-rest field encryption.
- Embedded in `hayson-server`, `hayson-cli` (local mode) and `hayson-browser`; shares its on-disk
format with the TypeScript `@theritual/hayson` package.