FWOB

FWOB is a Rust implementation of the Fixed-Width Ordered Binary format family.

The project provides two format versions:

1. FWOB v1 for compact fixed-width ordered files.
2. FWOB v2, a fixed-page compressed format for high-performance random access, range queries, and bulk append workloads.

FWOB v2 keeps page addresses arithmetic while allowing each page to contain a variable number of fixed-width frames. A page is a fixed-size on-disk container with an 80-byte header, compressed payload, and zero padding.

Workspace

• fwob-core: shared schema, frame, key, reader/writer handles, service traits, and error types.
• fwob-derive: derive macro for strongly typed fixed-width frames.
• fwob-v1: FWOB v1 reader, writer, verifier, and compatibility tests.
• fwob-v2: compressed fixed-page FWOB v2 reader and writer.
• fwob: the primary library facade with auto-detecting Reader, Writer, Editor, Maintenance, and Organizer APIs, plus the command-line tool.

The logical Rust API is documented in docs/api.md. Repair can promote complete ordered frames or pages left beyond the committed count by an interrupted write, while truncating partial or invalid tails.

Installation

Install the command-line tool from crates.io:

cargo install fwob

Library crates are available separately as fwob, fwob-core, fwob-derive, fwob-v1, and fwob-v2.

Library Quick Start

use fwob::{Reader, Writer};
use fwob_core::{Field, FieldType, Schema};

let schema = Schema::new(
    "Tick",
    vec![Field::new("time", FieldType::SignedInteger, 8, 0)],
    0,
)?;

let mut writer = Writer::create_v2(
    "ticks.fwob",
    schema,
    fwob_v2::WriterOptions::new("prices"),
)?;
writer.append_frame(&123_i64.to_le_bytes())?;
writer.finish()?;

let mut reader = Reader::open("ticks.fwob")?;
let first = reader.first_frame()?.expect("one frame");
assert_eq!(first.bytes(), 123_i64.to_le_bytes());
# Ok::<(), Box<dyn std::error::Error>>(())

See docs/api.md for reading, appending, editing, maintenance, organization, typed-frame, and format-specific examples.

Typed frames map ordinary Rust structs directly to the stored schema:

use fwob::{TypedEditor, TypedReader, TypedWriter};
use fwob_core::FwobFrame;

#[derive(Debug, PartialEq, FwobFrame)]
struct Tick {
    #[fwob(key)]
    time: i64,
    price: u32,
    size: i32,
}

let mut writer = TypedWriter::<Tick>::create_v2(
    "ticks.fwob",
    fwob_v2::WriterOptions::new("prices"),
)?;
writer.append(&Tick { time: 1, price: 500, size: 10 })?;
writer.finish()?;

let mut reader = TypedReader::<Tick>::open("ticks.fwob")?;
assert_eq!(reader.read_frame(0)?.unwrap().price, 500);

let mut editor = TypedEditor::<Tick>::open("ticks.fwob")?;
editor.delete_ranges(&[0..1])?;
# Ok::<(), fwob::Error>(())

Fixed-width UTF-8 fields can use fwob_core::FixedString<N>. Values are space-padded to exactly N bytes and rejected when their encoded byte length exceeds the declared width. The typed API also re-exports fwob_core::Decimal with the legacy 16-byte decimal representation. Ordered keys may be integers, f32, f64, or Decimal. String-table fields may use StringIndex8, StringIndex16, StringIndex (32-bit), or StringIndex64. On v2, integer fields may declare Unix epoch semantics with #[fwob(timestamp = "seconds")] or the millisecond, microsecond, and nanosecond variants. Table and Markdown output render those fields as UTC.

Command Examples

fwob verify ticks.fwob
fwob inspect ticks.fwob
fwob convert ticks.fwob ticks-v2.fwob smallest 1MiB --zstd-level 9
fwob convert ticks.fwob ticks-columnar.fwob columnar-basic zstd
fwob convert v2 ticks.fwob ticks-delta.fwob columnar-delta zstd verify
fwob append ticks-v2.fwob new-ticks.fwob verify
fwob split ticks.fwob parts 1000 2000 3000 zstd columnar-basic
fwob concat ticks-joined.fwob parts/ticks.part0.fwob parts/ticks.part1.fwob zstd
fwob edit ticks-joined.fwob --title Renamed --append-string NASDAQ
fwob find ticks-v2.fwob 100..200
fwob find ticks-v2.fwob 100 200..300 500.. ..50
fwob dump ticks-v2.fwob 100 200..300 csv
fwob dump ticks-v2.fwob raw > ticks.txt
fwob delete ticks-v2.fwob 100 200 local-repack verify
fwob delete ticks-v2.fwob 100 200 repack-to-end zstd columnar-basic compress-partial-page
fwob verify ticks-v2.fwob
fwob bench range ticks-v2.fwob --first-key-i32 100 --last-key-i32 200

Positional tokens are case-sensitive. For example, v2, zstd, and 1MiB are tokens; V2, ZSTD, and 1MIB are treated as paths or values rather than their lowercase token forms.

Tuning Parameters