# Value-format migrations
Value-format migrations evolve stored **value bytes** before ArmDB opens a
typed collection. They are for changes that the new codec cannot decode from
the old bytes: changed field types, removed required fields, codec changes, or
a changed fixed value width in a `ZeroTree` or `ZeroMap`.
They do not migrate keys, collection names, transaction-log bytes, wire
formats, or LSM storage. They are eager physical rewrites, not lazy value
envelopes.
## Choosing a transition
Advance `CollectionMeta::VERSION` one version at a time. Every stored version
transition needs exactly one handler:
| Old bytes cannot decode as the new type | `FORMAT_MIGRATIONS` | Rewrites files before typed recovery. |
| Old bytes decode as the new type, but the value changes | existing typed value migration | Updates through the typed layer after open. |
| Bytes stay valid and only the schema interpretation changes | typed value migration returning `MigrateAction::Keep` | Updates version/type metadata without rewriting value bytes. |
Use the cheapest correct choice. A format migration is required whenever the
new type cannot read the old bytes; a typed migration cannot repair a decode
failure because decoding happens before that callback runs.
Format steps must be a contiguous prefix. `1 -> 2` format followed by `2 -> 3`
value is valid; value followed by format is rejected with
`FormatStepAfterValueStep`.
## Declaring format steps
`CollectionMeta::FORMAT_MIGRATIONS` defaults to an empty slice. Declare a
format prefix next to `NAME` and `VERSION`:
```rust
const FORMAT_MIGRATIONS: &'static [armdb::FormatMigration] = &[
armdb::format_step!(1, v1::User => User, RapiraCodec, |old| {
Ok::<_, armdb::DbError>(Some(User {
name: old.name,
age: None,
}))
}),
];
```
`format_step!` has four forms:
```rust
// One codec, callback sees only the decoded old value.
format_step!(from, Old => New, Codec, |old| body)
// Separate old and new codecs.
format_step!(from, Old => New, OldCodec => NewCodec, |old| body)
// One codec, and the immutable original key bytes are available.
format_step!(from, Old => New, Codec, |key, old| body)
// Separate codecs with the key-aware callback.
format_step!(from, Old => New, OldCodec => NewCodec, |key, old| body)
```
Each body returns `DbResult<Option<New>>`: `Some` encodes and writes the new
bytes; `None` drops that row. `ZeroTree` and `ZeroMap` use the same forms with
`ZerocopyCodec`.
For a byte-level decision that must avoid decoding, use a raw step:
```rust
fn preserve(key: &[u8], old: &[u8], out: &mut Vec<u8>) -> armdb::DbResult<armdb::FormatAction> {
if key.starts_with(b"archive/") {
return Ok(armdb::FormatAction::Drop);
}
out.extend_from_slice(old);
Ok(armdb::FormatAction::Write)
}
const STEP: armdb::FormatMigration = armdb::FormatMigration::raw_checked(
1, &<Old as armour_core::GetType>::TYPE, &<New as armour_core::GetType>::TYPE, preserve,
);
```
`FormatAction::Keep` retains the input bytes exactly; `out` is ignored for
`Keep` and `Drop`. The runner clears `out` before every callback and reuses two
buffers per shard. `FormatMigration::raw` omits type checks; prefer
`raw_checked` whenever the endpoints are known.
## Validation and callback contract
Before any migration state or files are written, ArmDB validates the combined
format and typed-value chain. It rejects duplicate handlers for a version with
`DuplicateMigrationStep`, missing transitions with `SchemaMismatch`, and a
format step after a value step with `FormatStepAfterValueStep`.
Checked format links must form a type chain from the persisted type hash to the
current collection type. A mismatch is `FormatTypMismatch`. A raw link skips
only its local type check and adds a warning to the report. On first open there
is no stored data to migrate; at the same version the normal type-hash guard
still applies.
Callbacks may be non-idempotent: a shard swap means a completed target shard is
not transformed again after a crash. They must nevertheless be deterministic.
For identical key and value bytes, produce identical output without clocks,
randomness, network calls, or other external state.
## Automatic and explicit execution
Both `Config` and `FixedConfig` default to `FormatMigrationPolicy::Auto`, which
runs the format prefix during the managed `open_*` path. To schedule the
rewrite separately, configure `Refuse`:
Only the managed `Db::open_*` recipes and the explicit `Db::migrate_format*`
methods execute `FORMAT_MIGRATIONS`. Direct collection constructors such as
`TypedTree::open_hooked` have no `Db` metadata or migration coordinator and
therefore ignore the declaration; do not use them to open old bytes with a new
codec or layout.
```rust
let config = armdb::Config::balanced()
.format_migration(armdb::FormatMigrationPolicy::Refuse)
.build();
```
An open that needs a format rewrite then returns
`FormatMigrationRequired { collection, from, to }` without writes. Run the
physical prefix while the collection is not open on that `Db`:
```rust
let report = db.migrate_format::<User>(&config)?; // Bitcask
let report = db.migrate_format_fixed::<Session, 32>(&fixed_config)?; // FixedStore Zero*
```
The explicit methods reserve the collection name and fail with the existing
duplicate-collection protection if it is live. They perform only the physical
format prefix. The following managed open performs pending-transaction replay,
remaining typed value migrations, and authoritative projection/counter repair.
`FormatMigrationPolicy::Refuse` applies only to managed opens: the explicit
`migrate_format*` methods are the scheduled executor and intentionally bypass
that policy after validating the supplied configuration.
The explicit signatures do not receive the collection hook or a schema
registry, so they cannot prove that a source has no unique claims. Every
non-empty explicit rewrite conservatively sets `unique_dirty` and adds an
operator warning. A registry-aware `schema::repair` is the only retirement
path. An application built without schema repair should use automatic managed
migration when its hook can prove that the source has no claims; it must not
treat an explicit marker as self-clearing.
Keep the declared format chain until every pending transaction encoded under an
accepted old fingerprint has been checkpointed. Pending records are retained
unchanged on disk and transformed only in memory during replay; removing the
chain early can produce `TxSchemaMismatch` or make recovery impossible.
This retention rule still applies after an explicit rewrite has completed and
after a crash past the format metadata commit: the installed collection version
does not prove that every immutable pending transaction record has replayed.
A transaction record qualifies its value bytes with the stored type hash, not a
codec or format-version identifier. A checked format prefix that maps the same
type hash to more than one version (for example, a codec-only migration of the
same Rust type), or a raw prefix without checked type hashes, is therefore
ambiguous while a matching transaction remains pending. Raw history is
ambiguous even when the logged fingerprint equals the current target type: that
equality does not identify which raw version produced the bytes. Both automatic
and explicit migration perform this admission before dirty flags, migration
markers, file-id reservations, or shard rewrites and return
`TxFormatAmbiguous`. ArmDB never writes unqualified old bytes directly into the
new codec. This conservative barrier may require checkpointing records whose
operation already landed when another participant keeps the transaction
pending; the safe normal procedure is to drain/checkpoint the txlog before the
format rewrite. As an operator-only recovery escape hatch,
`TxConfig::discard_pending_on_schema_mismatch = true` marks every affected
transaction record applied and durably checkpoints the txlog before the rewrite.
That setting can discard sibling operations or freeze a transaction after only
an earlier-opened sibling applied, so leave it off during normal upgrades.
Typed value migrations have a pre-existing limitation: they do not change the
persisted `typ_hash`. If bytes logged before and after a value migration share
that hash, replay cannot distinguish those two value versions. Drain and
checkpoint the transaction log before such a transition; format migration does
not alter the transaction record format to solve this separate ambiguity.
## Supported matrix
| `TypedTree`, `TypedMap` | supported | not available |
| `VarTypedTree`, `VarTypedMap` | supported | not available |
| `ZeroTree`, `ZeroMap` | supported | supported |
| `LsmTree` | rejected when the format chain is non-empty | n/a |
`LsmTree` returns `FormatMigrationUnsupported { collection, backend: "lsm" }`.
There is no generic `armdb-cli` command: applications must provide the type
declarations and codecs for their own format chain.
## Rewrite and recovery behavior
The migration rewrites each shard independently, preserving live-row GSNs.
It carries no tombstones or superseded rows, and a dropped row is absent from
the target. At most `min(shard_count, available_parallelism())` shards are
rewritten concurrently. Plan disk capacity for at most
`size_of_one_shard * in_flight_shards` of additional data, plus normal filesystem
overhead; this is a shard-atomic rather than collection-sized temporary bound.
The rewrite is irreversible once the swap removes `.old`. ArmDB does not decode
every rewritten target value before advancing `db.info`; a deterministic but
incorrect callback can therefore publish unreadable bytes. Take and verify a
backup before deployment and keep it until the migrated application has opened
and exercised the target collection successfully.
Bitcask live-set selection retains keys, headers, and disk locations, then
rereads each winning value for transformation. The second read is deliberate:
retaining all live values would make migration memory proportional to an entire
large-value shard rather than to selection metadata.
Each target shard gets a CRC-protected `format.v` before its atomic directory
swap. The marker itself is published as temp write, file fsync, rename, and
directory fsync; a failed publication never truncates the last durable marker.
A collection `.migration` marker describes interrupted work, but shard state is
authoritative: restart resolves completed targets, stale temporary outputs,
missing live shards, and source shards without applying a completed
non-idempotent step again. A legacy/torn marker in `.mig` is disposable when a
recoverable source copy still exists as the live shard or `.old`. The collection
marker is removed only after `db.info` has committed the new metadata.
Encrypted Bitcask output uses file ids strictly greater than every file id ever
used or durably reserved in the source shard. Before the first target-key page
is sealed, the rewrite persists a CRC-protected per-shard high-water floor for
the whole output attempt using temp write, file fsync, rename, and directory
fsync. A failed or discarded `.mig` therefore burns those ids; retrying with a
different binary or different plaintext advances to a new range. An ordinary
encrypted shard open also admits that floor into `next_file_id`, so continuing
to write the recoverable old collection after a pre-swap failure cannot reuse a
burned migration nonce. This prevents AES-GCM nonce reuse because a sibling
`.mig` directory is not a new encryption domain. The rewrite preflights the
needed ids and returns `FormatFileIdExhausted { collection, shard, needed }`
instead of wrapping. Rotate to a new key/salt domain before retrying an exhausted
shard.
Transaction-log files are never rewritten or re-sealed.
FixedStore rebuilds every physical slot at the same slot id. An occupied slot
whose CRC is torn is handled exactly like ordinary dirty recovery: its version
bits are preserved, the target slot is marked free, the pass continues, and the
report contains an operator warning with the aggregate recovered-slot count.
Structural header, width, shard-identity, and file-length corruption remain hard
errors.
## Repair, TTL, watches, and replication
The format phase sets durable `projections_dirty` before backend dispatch.
After open and replay, the normal projection reconciliation removes stale index
rows, including rows for `Drop`. It marks counters dirty, and the existing
recount repairs them. Claim-bearing sources also retain `unique_dirty` until a
registry-aware namespace repair succeeds; `schema::validate` reports that flag
as an error, and `Db::unique_dirty(name)` lets an application keep traffic
closed until repair.
The raw rewrite emits no watch or change-stream events. TTL indexes are rebuilt
after open from the final migrated values. The pass issues no new GSNs and
persists collection and per-shard high-water floors so post-migration writes and
pending-transaction replay cannot reuse or misclassify an old GSN.
A format change is a coordinated follower upgrade: every follower must run code
that understands the target format, and nodes migrate their own files before
participating normally. Upgrade with followers caught up. Re-seed a follower
that has fallen too far behind, because the rewrite drops historical versions
and tombstones just as compaction does.
## Reports and errors
An explicit run returns `FormatMigrationReport`:
| `collection` | Static collection name. |
| `from`, `to` | Stored source version and completed format-prefix version. |
| `shards` | Number of rewritten shards. |
| `rows_migrated` | Live rows written to target shards. |
| `rows_dropped` | Live rows removed by `Drop`. |
| `bytes_before`, `bytes_after` | Aggregate source and target shard bytes. |
| `elapsed` | Duration of the physical pass. |
| `warnings` | Raw-link/type-suffix, explicit-repair, and recovered torn FixedStore-slot warnings that require operator attention. |
The format-specific errors and their fields are:
| `FormatMigrationRequired` | `collection`, stored `from`, required format-prefix `to`. |
| `DuplicateMigrationStep` | `collection`, duplicated source `version`. |
| `FormatStepAfterValueStep` | `collection`, invalid format-step `version`. |
| `FormatTypMismatch` | `collection`, transition `from`, expected type hash `expected`, observed hash `stored`. |
| `FormatStepFailed` | `collection`, source `from_version`, failing row `key` encoded as hex, callback `cause`. |
| `FormatFileIdExhausted` | `collection`, `shard`, number of fresh ids `needed`. |
| `FormatMigrationUnsupported` | `collection`, rejected `backend`. |
| `TxFormatAmbiguous` | `collection`, pending `tx_id`, unqualified type `fingerprint`; no rewrite occurred. |
Ordinary validation, filesystem, encryption, codec, recovery, and transaction
errors remain their existing `DbError` variants.