Skip to main content

Crate bizstd

Crate bizstd 

Source
Expand description

bizstd — “Blazing-fast Insert ZSTD compressed file format”.

A self-describing binary container: a fixed 16-byte preamble, a fixed-size text header zone updated in place, and a data section of independent zstd frames followed by an uncompressed crash-safe tail.

preamble (16 bytes, binary):
  magic "BIZSTD"   6 bytes
  version          u8
  flags            u8      (bit 0: data compressed)
  header_area_len  u32 LE  (usually 4096)
  reserved         4 bytes

header zone (ASCII text inside header_area_len bytes):
  key:value\n      key `_?[a-z0-9][a-z0-9_]*`, split on the FIRST ':'
  \n               an empty line ends the headers
  zero padding to the end of the zone

data (at offset 16 + header_area_len):
  zstd frames of closed hours, then the raw uncompressed tail of the
  hour being written

System keys start with _ and are a closed list owned by this file: _schema, _schema_fields, _schema_hash, _record, _source, _created_at, _writer, _compression, _compression_level, _records, _bytes_raw, _frames, _preview, _sealed. Every one of them is a rebuildable cache — the data section is the truth. The counters _records and _bytes_raw cover closed frames only; records still sitting in the raw tail join them when their frame closes. Application keys must NOT start with _; the container never interprets them.

Values escape exactly two characters: \n and \\. Header values built by this file contain neither, so their round trip is byte-identical.

Closing a frame is crash-safe through a redo journal (<file>.seal): the compressed frame and the new header zone become durable in the journal first, then the raw tail is overwritten in place, the file is truncated and the zone rewritten, and only then the journal is removed. Container::open_append replays a complete journal and discards an incomplete one, then truncates a torn record off the raw tail — a crash at any instant loses at most the buffered records of the current write.

The crate is deliberately small — std and zstd only, no other dependency and nothing to configure — so that reading a container needs this crate and nothing else.

§Concurrency

One writer per file, and enforcing that is the caller’s job. Nothing here takes a lock. Two handles opened with Container::open_append on the same path will both recover the tail, both buffer records, and both write at offsets the other does not know about; the file that results is not one either of them wrote. There is no error for this and no detection — only the guarantee that it will not be noticed until the data is read back.

Readers are unrestricted among themselves: Container::open_read never writes. A reader running alongside a writer is safe in the sense that it cannot corrupt anything, but what it sees is a file in motion — the tail grows under it, and a frame being closed changes both the data and the header zone. Reading a file that is being written to gives a consistent answer only for the frames already closed when the read started.

§Example

use bizstd::{Container, DEFAULT_HEADER_AREA, FieldSpec, HOT_LEVEL, RecordLayout, Schema};

let schema = Schema {
    name: "samples@1".to_owned(),
    fields: vec![
        FieldSpec { name: "time_nanos".to_owned(), ty: "u64".to_owned(), offset: 0 },
        FieldSpec { name: "value".to_owned(), ty: "f64".to_owned(), offset: 8 },
    ],
    layout: RecordLayout::Fixed(16),
};

let mut file = Container::create(
    "samples.bizstd".as_ref(),
    &schema,
    "example",
    "example writer",
    0,
    DEFAULT_HEADER_AREA,
    &[("stream", "alpha")],
)?;

let mut record = [0u8; 16];
record[..8].copy_from_slice(&1_u64.to_le_bytes());
record[8..].copy_from_slice(&2.5_f64.to_le_bytes());
file.append_record(&record)?;
file.seal(0, HOT_LEVEL)?;

Structs§

Container
A .bizstd file, open for reading or appending.
FieldSpec
One field of a record, for _schema_fields.
Frame
One closed zstd frame in the data section.
Headers
The ordered key:value pairs of the header zone.
Preamble
The 16-byte binary preamble.
RebuildReport
What rebuild_headers found and did.
RepackReport
What repack measured.
Schema
What the records are: a name, the fields, and the boundary rule.
ValidateReport
What validate found.

Enums§

Error
Everything this crate can fail with.
FieldValue
One field of one record, decoded according to the schema.
RecordLayout
How record boundaries are found in the raw byte stream.

Constants§

COLD_LEVEL
The zstd level for offline repacking of cold files.
DEFAULT_HEADER_AREA
The default header zone size.
EXTENSION
The file extension, without the dot.
FLAG_COMPRESSED
Preamble flag bit 0: the data section carries zstd frames.
HOT_LEVEL
The zstd level for closing frames on the hot path.
MAGIC
The preamble magic, exactly six bytes.
MAX_FRAME_RAW_BYTES
The largest raw size one frame may decompress to. Without a ceiling a hostile file compresses a few kilobytes into a demand for all the memory there is.
MAX_HEADER_AREA
The largest header zone a file may declare. A u32 read straight from a 20-byte file would otherwise ask the allocator for 4 GiB, and an allocation that cannot be served aborts the process rather than returning an error.
NO_COMPRESSION
No compression at all: frames are stored as the bytes that were appended.
PREAMBLE_BYTES
The preamble’s fixed size.
VERSION
The container version this file writes.

Functions§

escape_value
Escapes the two characters a value may not carry raw: \n and \.
fnv1a64
FNV-1a 64 — the schema fingerprint. A fingerprint, not a signature.
key_is_valid
Whether the text is a legal header key: _?[a-z0-9][a-z0-9_]*.
max_frames_for
Roughly how many frames a header zone of header_area bytes can list.
parse_frames
Parses _frames. The fourth field, the frame checksum, is optional: files written before it existed carry triples.
peek_headers
Reads just the preamble and header zone — the cheap peek the catalogue does per file, one bounded read, never touching the data section.
read_field
Reads one field out of one record.
rebuild_headers
Rescans the data section by walking zstd frames, recomputes _frames, _records and _bytes_raw, and rewrites the zone when fix is set.
render_frames
Renders _frames: id,offset,len,xxh64;... (hash in hex; the fourth field is omitted while unrecorded, so an index written before checksums existed round-trips byte-for-byte).
repack
Re-encodes every closed frame at level, transactionally per file.
repack_with_header_area
Re-encodes every closed frame at level and gives the rebuilt file a header zone of header_area bytes.
rfc3339_utc
RFC 3339 UTC from epoch milliseconds — civil-from-days, Hinnant.
split_records
Splits raw bytes into records; the second value is the torn leftover.
unescape_value
Reverses escape_value.
validate
Checks a file: headers, schema hash, frame decode, record boundaries, counters. check receives every record and returns a problem or None; pass None to skip content checks. At most 20 problems are collected.
xxh64
XXH64 of data with seed 0 — the per-frame checksum.

Type Aliases§

Result
The result type every fallible function in this crate returns.