Expand description
bizstd — “Blazing-fast Insert ZSTD compressed file format”.
A self-describing binary container: a fixed 16-byte preamble, a fixed-size text header zone updated in place, and a data section of independent zstd frames followed by an uncompressed crash-safe tail.
preamble (16 bytes, binary):
magic "BIZSTD" 6 bytes
version u8
flags u8 (bit 0: data compressed)
header_area_len u32 LE (usually 4096)
reserved 4 bytes
header zone (ASCII text inside header_area_len bytes):
key:value\n key `_?[a-z0-9][a-z0-9_]*`, split on the FIRST ':'
\n an empty line ends the headers
zero padding to the end of the zone
data (at offset 16 + header_area_len):
zstd frames of closed hours, then the raw uncompressed tail of the
hour being writtenSystem keys start with _ and are a closed list owned by this file:
_schema, _schema_fields, _schema_hash, _record, _source,
_created_at, _writer, _compression, _compression_level,
_records, _bytes_raw, _frames, _preview, _sealed. Every one of
them is a rebuildable cache — the data section is the truth. The
counters _records and _bytes_raw cover closed frames only; records
still sitting in the raw tail join them when their frame closes.
Application keys must NOT start with _; the container never
interprets them.
Values escape exactly two characters: \n and \\. Header values built
by this file contain neither, so their round trip is byte-identical.
Closing a frame is crash-safe through a redo journal (<file>.seal):
the compressed frame and the new header zone become durable in the
journal first, then the raw tail is overwritten in place, the file is
truncated and the zone rewritten, and only then the journal is removed.
Container::open_append replays a complete journal and discards an
incomplete one, then truncates a torn record off the raw tail — a crash
at any instant loses at most the buffered records of the current write.
The crate is deliberately small — std and zstd only, no other
dependency and nothing to configure — so that reading a container needs
this crate and nothing else.
§Concurrency
One writer per file, and enforcing that is the caller’s job. Nothing
here takes a lock. Two handles opened with Container::open_append on
the same path will both recover the tail, both buffer records, and both
write at offsets the other does not know about; the file that results is
not one either of them wrote. There is no error for this and no detection —
only the guarantee that it will not be noticed until the data is read back.
Readers are unrestricted among themselves: Container::open_read never
writes. A reader running alongside a writer is safe in the sense that it
cannot corrupt anything, but what it sees is a file in motion — the tail
grows under it, and a frame being closed changes both the data and the
header zone. Reading a file that is being written to gives a consistent
answer only for the frames already closed when the read started.
§Example
use bizstd::{Container, DEFAULT_HEADER_AREA, FieldSpec, HOT_LEVEL, RecordLayout, Schema};
let schema = Schema {
name: "samples@1".to_owned(),
fields: vec![
FieldSpec { name: "time_nanos".to_owned(), ty: "u64".to_owned(), offset: 0 },
FieldSpec { name: "value".to_owned(), ty: "f64".to_owned(), offset: 8 },
],
layout: RecordLayout::Fixed(16),
};
let mut file = Container::create(
"samples.bizstd".as_ref(),
&schema,
"example",
"example writer",
0,
DEFAULT_HEADER_AREA,
&[("stream", "alpha")],
)?;
let mut record = [0u8; 16];
record[..8].copy_from_slice(&1_u64.to_le_bytes());
record[8..].copy_from_slice(&2.5_f64.to_le_bytes());
file.append_record(&record)?;
file.seal(0, HOT_LEVEL)?;Structs§
- Container
- A
.bizstdfile, open for reading or appending. - Field
Spec - One field of a record, for
_schema_fields. - Frame
- One closed zstd frame in the data section.
- Headers
- The ordered key:value pairs of the header zone.
- Preamble
- The 16-byte binary preamble.
- Rebuild
Report - What
rebuild_headersfound and did. - Repack
Report - What
repackmeasured. - Schema
- What the records are: a name, the fields, and the boundary rule.
- Validate
Report - What
validatefound.
Enums§
- Error
- Everything this crate can fail with.
- Field
Value - One field of one record, decoded according to the schema.
- Record
Layout - How record boundaries are found in the raw byte stream.
Constants§
- COLD_
LEVEL - The zstd level for offline repacking of cold files.
- DEFAULT_
HEADER_ AREA - The default header zone size.
- EXTENSION
- The file extension, without the dot.
- FLAG_
COMPRESSED - Preamble flag bit 0: the data section carries zstd frames.
- HOT_
LEVEL - The zstd level for closing frames on the hot path.
- MAGIC
- The preamble magic, exactly six bytes.
- MAX_
FRAME_ RAW_ BYTES - The largest raw size one frame may decompress to. Without a ceiling a hostile file compresses a few kilobytes into a demand for all the memory there is.
- MAX_
HEADER_ AREA - The largest header zone a file may declare. A
u32read straight from a 20-byte file would otherwise ask the allocator for 4 GiB, and an allocation that cannot be served aborts the process rather than returning an error. - NO_
COMPRESSION - No compression at all: frames are stored as the bytes that were appended.
- PREAMBLE_
BYTES - The preamble’s fixed size.
- VERSION
- The container version this file writes.
Functions§
- escape_
value - Escapes the two characters a value may not carry raw:
\nand\. - fnv1a64
- FNV-1a 64 — the schema fingerprint. A fingerprint, not a signature.
- key_
is_ valid - Whether the text is a legal header key:
_?[a-z0-9][a-z0-9_]*. - max_
frames_ for - Roughly how many frames a header zone of
header_areabytes can list. - parse_
frames - Parses
_frames. The fourth field, the frame checksum, is optional: files written before it existed carry triples. - peek_
headers - Reads just the preamble and header zone — the cheap peek the catalogue does per file, one bounded read, never touching the data section.
- read_
field - Reads one field out of one record.
- rebuild_
headers - Rescans the data section by walking zstd frames, recomputes
_frames,_recordsand_bytes_raw, and rewrites the zone whenfixis set. - render_
frames - Renders
_frames:id,offset,len,xxh64;...(hash in hex; the fourth field is omitted while unrecorded, so an index written before checksums existed round-trips byte-for-byte). - repack
- Re-encodes every closed frame at
level, transactionally per file. - repack_
with_ header_ area - Re-encodes every closed frame at
leveland gives the rebuilt file a header zone ofheader_areabytes. - rfc3339_
utc - RFC 3339 UTC from epoch milliseconds — civil-from-days, Hinnant.
- split_
records - Splits raw bytes into records; the second value is the torn leftover.
- unescape_
value - Reverses
escape_value. - validate
- Checks a file: headers, schema hash, frame decode, record boundaries,
counters.
checkreceives every record and returns a problem orNone; passNoneto skip content checks. At most 20 problems are collected. - xxh64
- XXH64 of
datawith seed 0 — the per-frame checksum.
Type Aliases§
- Result
- The result type every fallible function in this crate returns.