data_bucket 0.7.11

DataBucket is container for WorkTable's data
Documentation

DataBucket

Page framing, row directories and index storage for WorkTable.

Version 3 cutover

DataBucket 0.7 writes page format 3. Older page formats are rejected with a version error; opening them does not convert or remove them. Deployments that can regenerate their data should explicitly recreate the store. Retained data requires an application-specific conversion with the old reader.

For a page of P bytes, the general header occupies the first 28 bytes. Data pages contain row bytes followed by a live-row directory at the page tail. Each directory entry is a little-endian (u32 offset, u32 length) pair; its offset is relative to the payload. A CRC-32 is stored at P-8 and the entry count at P-4. The checksum covers the entire payload except its own word, including padding, directory and count. Header identity and version are validated separately. Index and metadata pages retain their existing body layouts, with the new page-version marker.

data_page_row_capacity computes a row allocation budget that reserves the maximum directory space for the minimum archived row size. DataPage::encode and decode validate directory extents and integrity. Updates, deletion and relocation must maintain DataPage::rows; raw row bytes alone are not a complete v3 persisted data page. The file-level update_at accepts a readable and writable file, validates the existing live row and updates the checksum.

The table schema version and crate package version are separate from the page-format version. WorkTable's Vec snapshot files use a different container and cannot be treated as ordinary DataBucket space files.

CRC validation detects damaged data pages; it does not provide a transaction log, atomic multi-file commits or crash repair. WorkTable owns synchronization and durability policy.

Storage domains

storage::StorageDomain groups every table and index page in one database-wide generation. begin_generation() returns a builder for page puts, page deletes and semantic catalog changes. commit_generation() stages immutable page objects, prepares the system catalog, writes its checkpoint, conditionally advances the small bootstrap head, and only then publishes the prepared catalog in memory. A failed stage or conflicting head leaves the preceding catalog visible.

DataBucket defines the bounded catalog records and the SystemCatalog provider interface. WorkTable supplies the provider as a generated table. This keeps the Cargo dependency one-way while DataBucket retains the only catalog write permit at runtime. Applications query WorkTable's read-only catalog view.

storage::PageStore supplies restart, range-read, stage and conditional commit operations. With s3-support, storage::s3::S3PageStore implements that contract for Tigris and compatible S3 services. Data pages are stored in content-addressed segments. Adjacent dirty pages may coalesce up to 4 MiB, but the target is never a minimum. One changed page remains one page-sized data upload. S3 responses and reconstructed catalog checkpoints are capped at 64 MiB. Remote endpoints must use HTTPS; plain HTTP is accepted only for loopback test servers.

The generated catalog checkpoint is also divided into content-addressed PAGE_SIZE chunks. A mutation uploads only changed catalog chunks, a compact chunk manifest and the bootstrap head. Catalog rows use stable tombstones so a delete does not shift every later row and turn a small mutation into a complete catalog upload. StorageDomain::open restores the committed catalog before any page lookup, and read_page resolves a logical page through that catalog and issues one bounded range read.

The library is no_std with alloc and uses Nagoya's portable I/O traits. The validate-reads compatibility feature remains enabled by default; safe archived access validates bytes regardless of feature selection. The library requires an allocator. The concurrent index dependency uses OS services through libc on supported targets; no_std does not imply a bare-metal implementation. The command line tools and host tests use std.

Legacy snapshot compatibility

storage::legacy_snapshot::LegacySnapshotStore owns the legacy WTS3M001 and WTS3M002 manifest formats, S3 transport, immutable chunks and staged directory restoration. WorkTable delegates to this backend instead of owning object-store code. This is distinct from StorageDomain generations: opening a legacy snapshot does not migrate its format. Ordinary opens fetch immutable segments with bounded concurrency and verify them before installing the staged directory. After initializing local table files, consumers call capture_baseline before applying mutations and sync after persistence completes.

This compatibility backend does not reclaim superseded objects. Do not expire chunks by age: old chunks may remain referenced. Reclamation needs storage-layer coordination with writers, restore readers and retained recovery roots; record retention remains an application policy.

Append-only retention segments

With s3-support, storage::stream::AppendStream supplies a bounded, opaque-byte append segment over StorageDomain. It seals every supplied image, even a short last page, and assigns new page ids on every append. No overwrite/delete method or mutable domain is exposed. The segment accepts up to 4096 pages of at most PAGE_SIZE bytes, then the caller rotates to another domain. Naming, record framing, deduplication and retention policy belong to the application. This module has no application-specific schema.

StreamCatalog uses a versioned DBSTR001 checkpoint with a 128-byte header and stable 128-byte page slots. It is a small SystemCatalog provider for opaque Metadata pages, independent of WorkTable's generated catalog. It does not alter existing archived layouts or DATA_VERSION. Unknown stream checkpoint versions are rejected. Existing table consumers are unchanged; validate-reads remains enabled by default.

S3 commits write new page segments, changed catalog chunks, the catalog manifest and the conditional head. For one new page and one changed catalog chunk this is four PUTs. S3PageStore::put_totals() reports attempted PUTs and body bytes, including retries, excluding headers and TLS overhead. Opening a domain loads its catalog chunks into the object cache, so unchanged chunks are reused after restart as well as within a process. Immutable objects use create-only PUTs; an existing object is verified rather than overwritten even when a cold writer meets identical page bytes.

S3PageStore::with_transport accepts an Arc<dyn S3Transport> with caller TLS and timeout policy; new retains the existing ureq transport. An async transport must execute on an isolated worker for this synchronous protocol. Missing ETags are rejected before a conditional update. Page range reads require HTTP 206 and verify length and checksum.

Before GC, retire(domain) conditionally installs a terminal head, including for an interrupted first generation. u64::MAX is a reserved writer epoch. Updated readers reject terminal heads before loading deleted catalogs, and commits cannot advance them. GC may then remove the domain's immutable objects, including unreferenced old manifests/chunks and staged orphans. Listing, deletion and GC retry/re-scan are caller responsibilities. Keep the terminal head until an application-enforced admission window and maximum in-flight request lifetime rule out delayed creation of generation one. Readers holding an old catalog must handle retention gaps.

The in-memory S3 transport tests specify exact first/second/reopened append PUT body counts, immutable-page preservation, range reads and terminal-head fencing. These tests require execution by the integrator before release.

Run sh scripts/check-no-std.sh -p data_bucket --lib to verify the default library graph with Rust std removed from the target sysroot. Build scripts and proc macros retain their normal host environment.

Command line tools

Create a sample file containing 2,500 records. An existing file is refused:

cargo run -p create-data-file -- --filename sample.wt.data --count 2500

Inspect page identities and live row extents without consulting an index:

cargo run -p dump-data-file -- --filename sample.wt.data
cargo run -p dump-data-file -- --filename sample.wt.data --hex

--count is a record count. The dumper reports each row's payload offset, archive length and absolute file offset. --hex includes its archive bytes; typed deserialization requires the owning application's row schema. For a nondefault stride, supply --page-size; supported values are 512, 4096, 8192, 16384 and 32768 bytes. The tools validate data-page checksums and directories.