Module formats 

Binary formats described by specs: one TOML file per format, on a search path. A spec names its format (acme.l2feed), says which files match (glob, magic, header values), and lays out header and records. Specs are data (no expressions or code: fields refer to earlier ones by name), and every size read from a file is bounded. Decoding is crate::formats::fixed_records’ for fixed-size records and crate::formats::framed_records’ otherwise; files reads a directory of one spec’s files. This module turns a spec and a file into that reader’s columns.

Modules

audio

WAV, BWF, RF64 and AIFF audio as a table of sample frames. The file is memory-mapped and a chunk walker finds the format and samples; row i is at data_offset + i * frame_bytes, so any window decodes from its own bytes (a 20 GB recording reads a screenful). Every stated length is checked against the file before use, so a hostile header errors rather than panics or over-allocates. The row count comes from the file size, letting a still-recording file grow ([AudioSource::extend]): recorders write a placeholder data size until they stop.

candump

CAN logs written by candump, decoded with DBC files.

column_types

A column’s declared type: one vocabulary for the type row, a delimited spec’s [columns], the typing of text columns, and a column retyped in the table.

columns

Rows built a row at a time into typed columns, handed over as a DataFrame.

csv_dialect

The parts of a CSV’s dialect that Polars’ reader does not have, named as the Frictionless Table Dialect names them.

dataflash

ArduPilot DataFlash logs (.bin).

dbc

DBC files: what the frames of a CAN bus mean.

delimited_spec

Format specs of kind = "delimited": the reading options for a family of CSV-like files, with header rows that have roles (names, units), a metadata line, and a few derived columns.

elf

ELF files as their symbol table: what uses flash and RAM.

excel

Excel workbooks: the sheets of one, listed on the home screen and summed up on the Info panel.

files

A directory of one spec’s files, {date}/{venue}/trades.bin: one table, the parts of each file’s path as columns.

fix

FIX logs: tag=value messages, SOH or | delimited, read into a table of one row per message.

fixed_records

Fixed-width values read straight out of a memory map. A column is values width bytes wide, stride apart, from start into a source; count side by side make an Array cell. Records (stride = record size) and one file per column (stride = width) share one strided decoder for format specs and NumPy; decode is public for other packed readers (audio, CAN). Sources are kept for raw-byte viewers. Frames decode over a row index (crate::formats::row_index) on the streaming engine; an untouched view’s window reads via FixedRecords::window without an index.

framed_records

Records not all one size, read from a memory map: length-prefixed records, variants picked by a type field, records found by a sync marker, records in compressed blocks or packet-capture payloads, and fixed records with fields read one at a time (varints, NUL-terminated text, counted groups, bit fields, deltas, checksums).

gps

GPS logs (NMEA 0183, GPX) as a table. Neither scans in place, so an open reads the file once in pieces, writing rows a batch at a time to a temp Arrow IPC file (claimed via the open’s Writer) that the dataset scans lazily. A GPX extension field appearing late starts a new segment file with the wider schema, joined with nulls; NMEA is always one segment. Readers are incremental, so a growing log can be followed later.

hf_splits

The splits of a Hugging Face datasets directory.

indexed

Tables of records an index found: the messages of one type in a flight log, the frames of one CAN message in a candump.

ipc_stream

Arrow IPC streams, the format Hugging Face datasets caches in: the IPC file format without the ARROW1 magic and footer, so Polars cannot scan it. An open converts the stream (or a directory’s stream shards) once into a temp IPC file and scans that; IPC files among the shards are scanned in place (Part). One record batch is held at a time: 0.2 GiB peak for a 3.0 GiB stream that read eagerly took 3.6 GiB.

journal

The systemd journal as journalctl -o json writes it: NDJSON whose records carry __CURSOR and __REALTIME_TIMESTAMP.

lines

Text read as it stands: a row per line, in a line column.

members

Files that hold several tables: a SQLite database’s tables and views, a NumPy archive’s arrays.

midi

Standard MIDI Files as a table of events. A hand-written parser (MThd/MTrk chunks, variable-length deltas, running status, sysex, meta events) checks every stated length against the remaining bytes, so a hostile file errors rather than panics or over-allocates. Rows are a small eager frame made lazy; the header, track names and tempo range go to a MidiSummary for Info’s MIDI tab.

model_files

SafeTensors and GGUF model files as a table of their tensors. Only the header is read, so a 70 GB checkpoint opens like a 7 MB one. Both parsers are hand-written over a bounded reader, checking every stated length before allocating or skipping, so a hostile header errors rather than allocating gigabytes or panicking. Rows are a small eager frame made lazy; metadata and totals go to a ModelSummary for Info’s Model tab.

nul_tail

A text file that ends in a run of NUL bytes ends where the run starts. Loggers that preallocate a file at a fixed size and never write past their last line leave the rest as NULs, which a CSV reader would take as one last row of junk text.

numpy

NumPy arrays: .npy files, and .npz archives of them.

parquet_footer

What a Parquet file’s footer says besides its rows: its row groups, each column’s compression and statistics, and who wrote it.

pushdown

Sources that read a window of rows themselves, and sources that filter and sort.

readers

The readers: what datui does with a file of each format.

row_index

Lazy frames decoded over a row index, for readers whose rows sit at places worked out from the row number: audio frames, fixed records, and the like.

schema_union

The schema of a dataset made of many files, which drift over time: every column any file has, from the footers the row count already reads, with one type per column:

sdf

SDF (structure-data format) compound files, read into a table: one row per record, with the molecule’s name, its atom and bond counts, and every data item (> <FIELD>) as a column.

sqlite

SQLite databases, read only.

text_formats

Text formats read into a table of their own: VCD value change dumps, FIX logs, SDF compound files and the GPS logs of crate::formats::gps; and Detail, every format’s tab of the Info panel.

ulog

PX4 ULog flight logs (.ulg).

vcd

VCD (value change dump) files from HDL simulators and logic analyzers, read into a long table: one row per value change of each signal.

Structs

Asked

What a request for a format says, besides the path.

BitField

One bit field of an integer: its own column.

BlockIndex

Blocks listed in the file, so their headers are not walked.

Blocks

Records in blocks, each with a header and, often, compressed on its own.

Capture

A packet capture whose payloads hold the records.

Checksum

A checksum over part of each record, or of the file before its footer.

Choice

Where a spec was chosen from, carried to a decompressed copy’s read.

DbcFound

A DBC file found on the search path.

Field

One field of a header or a record.

Files

A directory of one spec’s files: each file’s path is read by pattern.

FixFound

A FIX dictionary found on the search path, and the copies of the same name it hides.

Footer

Fields at the end of the file.

Found

A spec found on the search path, and the copies of the same name it hides.

Header

A spec’s header: its fields, and its size when that is more than they take.

HeaderValues

A header, read: each named field’s value, and how many bytes the header takes; and the footer’s values, and the symbol lists fields index into.

Lookup

Where a symbol list is, for a field of indexes into it.

MatchChip

One condition of a spec’s match, as home, Info and datui formats show it (magic MKTD, version 1, glob *.bin *.dat). Chips must all hold, except that a file the glob names is not asked its magic; alternatives live inside one chip.

Matched

The specs a file matches, by the first rule that matched any.

Opened

A file read through a spec: its columns, and what the read had to say.

PathPart

A part of a file’s path that becomes a column.

Read

A file read through a spec, as the open carries it to the dataset.

RecordType

One variant of a spec, as its documentation shows it.

Records

A spec’s records.

Registry

Every spec on the search path, first of each name first.

Section

A named run of the file that fields point into.

Spec

One format, as its spec describes it.

SpecDocs

What a spec says of its files, for the Documentation view: its own words and the notes its fields carry. None of it changes how a file is read.

SpecError

A problem with a spec: where it is (file, line, column) and what was expected.

Variant

One layout a record can take, picked by the type field.

Enums

Amount

A size or a count: written in the spec, or read from a field.

ChecksumAlgo

The checksums a spec can name.

ChipKind

What a chip’s value is, for its color and whether plain text quotes it.

Chosen

How a spec was chosen for a file.

Codec

The codec of each block: one for all, or one a header field names.

Compression

How a block’s body is compressed.

Delta

A value stored as the change from the previous record’s.

Encoding

How text is stored.

Endian

Byte order.

Expected

A header value a file must hold to match: match.where.

Framing

How one record is told from the next.

Layout

How the records sit in what is opened.

LookupFormat

How a symbol list’s entries are told apart.

Meaning

What a field means beyond its stored value.

Route

What route decided about one local path.

TimeUnitSpec

The unit a time field counts in.

Type

What a field’s bytes hold.

Constants

FORMAT_ONLY

What a spec with no match says in place of its conditions.

MAX_FLATTEN

The most columns one field may be flattened into.

MAX_SIZE

The largest size a spec or a file may give one field, header or record, and the most values one field may hold. A size read from a file past this is refused rather than believed.

MAX_SPEC_BYTES

The most a spec file may hold, local or remote: far more than any spec needs, and a bound on what --format FILE reads before it knows what it read.

MAX_SPEC_SAID

MAX_SPEC_BYTES, as the user is told it.

PATH_VAR

The variable the search path is extended with, after the config directory.

Traits

SpecRecords

The rows a spec reads, however they are framed: what the table, a window of it, formats check and the fuzz target read through.

Functions

all_fields

Every field of the records: the common ones, then each variant’s.

chips_plain

Chips as one line of plain text, for the command line and the Info panel: magic MKTD · version 1 · glob *.bin *.dat.

chosen_words

Why spec read a file, for the Notes tab: matched by magic MKTD · version 1 when its glob or its magic chose it, chosen by --format FILE otherwise.

command

datui formats, or datui formats check SPEC [FILE]: what to print, and the exit code (non-zero when the check finds an error).

head_of

The first reach bytes of path, through its decompressor when it has one.

is_spec_name

What a spec name may be: namespaced, as acme.l2feed is, so it cannot be taken for a built-in format.

path_parts

The parts {name} and {name:%Y%m%d} of a [files] path, in order.

read

Read path with the spec choice holds, naming it named in what it says.

route

Whether, and with which spec, path is read. In order: --format FILE, then --format NAME, then a glob, then magic. A file whose name or bytes say it is a format datui reads already keeps opening that way.

search_path

The directories and files searched for specs, in order: the config directory’s formats, then $DATUI_FORMATS_PATH, then [formats] path from the config.

search_path_for

The search path config asks for: the config directory’s formats, then $DATUI_FORMATS_PATH, then its [formats] path.

symbols

The entries of a symbol list.