.\" Generated by gen_docs from crates/datui-cli. Do not edit.
.TH DATUI\-FORMATS 7 2026-10-08 "datui 0.4.4" "Miscellaneous"
.nr HY 0
.ds AD l
.SH NAME
datui\-formats \- the formats datui reads, and format specs
.SH DESCRIPTION
datui reads 27 formats: Parquet, CSV, TSV, PSV, JSON, NDJSON, Arrow IPC, Avro, ORC, Excel, SafeTensors, GGUF, NMEA, GPX, WAV/AIFF audio, MIDI, SQLite, VCD, FIX, SDF, NumPy, ELF, ULog, DataFlash, candump, plain text, systemd journal, and binary formats you describe in a format spec.
.PP
The extension says the format; \fB\-\-format\fR names it when the extension does not, and text piped in is detected by content.
.PP
.RS 4
.EX
printf 'a,b\en1,2\en' > export.txt
datui \-\-format csv export.txt
.EE
.RE
.SH "HOW EACH FORMAT IS READ"
.TP
Parquet
\-\-format: \fBparquet\fR
.br
Extensions: \fB.parquet\fR
.br
Read: lazy scan
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: in place
.br
Bucket prefix: in place
.TP
CSV
\-\-format: \fBcsv\fR
.br
Extensions: \fB.csv\fR
.br
Read: lazy scan
.br
Compressed: decompressed copy
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: in place
.TP
TSV
\-\-format: \fBtsv\fR
.br
Extensions: \fB.tsv\fR
.br
Read: lazy scan
.br
Compressed: decompressed copy
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
PSV
\-\-format: \fBpsv\fR
.br
Extensions: \fB.psv\fR
.br
Read: lazy scan
.br
Compressed: decompressed copy
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
JSON
\-\-format: \fBjson\fR
.br
Extensions: \fB.json\fR
.br
Read: in memory
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
NDJSON
\-\-format: \fBjsonl\fR
.br
Extensions: \fB.jsonl\fR, \fB.ndjson\fR
.br
Read: in memory
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: in place
.TP
Arrow IPC
\-\-format: \fBarrow\fR
.br
Extensions: \fB.arrow\fR, \fB.arrows\fR, \fB.ipc\fR, \fB.feather\fR
.br
Read: lazy scan
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: in place
.br
Bucket prefix: in place
.TP
Avro
\-\-format: \fBavro\fR
.br
Extensions: \fB.avro\fR
.br
Read: in memory
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
ORC
\-\-format: \fBorc\fR
.br
Extensions: \fB.orc\fR
.br
Read: in memory
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
Excel
\-\-format: \fBexcel\fR
.br
Extensions: \fB.xls\fR, \fB.xlsx\fR, \fB.xlsm\fR, \fB.xlsb\fR
.br
Read: in memory
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
SafeTensors
\-\-format: \fBsafetensors\fR
.br
Extensions: \fB.safetensors\fR, \fB.safetensors.index.json\fR
.br
Read: in memory
.br
Compressed: no
.br
HTTP(S): in place
.br
In a bucket: in place
.br
Bucket prefix: in place
.TP
GGUF
\-\-format: \fBgguf\fR
.br
Extensions: \fB.gguf\fR
.br
Read: in memory
.br
Compressed: no
.br
HTTP(S): in place
.br
In a bucket: in place
.br
Bucket prefix: in place
.TP
NMEA
\-\-format: \fBnmea\fR
.br
Extensions: \fB.nmea\fR
.br
Read: converted to Arrow
.br
Compressed: converted to Arrow
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
GPX
\-\-format: \fBgpx\fR
.br
Extensions: \fB.gpx\fR
.br
Read: converted to Arrow
.br
Compressed: converted to Arrow
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
WAV, BWF, RF64, AIFF
\-\-format: \fBaudio\fR
.br
Extensions: \fB.wav\fR, \fB.wave\fR, \fB.bwf\fR, \fB.rf64\fR, \fB.aif\fR, \fB.aiff\fR, \fB.aifc\fR
.br
Read: lazy scan
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
MIDI
\-\-format: \fBmidi\fR
.br
Extensions: \fB.mid\fR, \fB.midi\fR, \fB.smf\fR, \fB.kar\fR, \fB.rmi\fR
.br
Read: in memory
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
SQLite
\-\-format: \fBsqlite\fR
.br
Extensions: \fB.db\fR, \fB.db3\fR, \fB.sqlite\fR, \fB.sqlite3\fR
.br
Read: lazy scan
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
VCD
\-\-format: \fBvcd\fR
.br
Extensions: \fB.vcd\fR
.br
Read: converted to Arrow
.br
Compressed: converted to Arrow
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
FIX
\-\-format: \fBfix\fR
.br
Extensions: none: by content
.br
Read: converted to Arrow
.br
Compressed: converted to Arrow
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
SDF
\-\-format: \fBsdf\fR
.br
Extensions: \fB.sdf\fR, \fB.sd\fR
.br
Read: converted to Arrow
.br
Compressed: converted to Arrow
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
NumPy
\-\-format: \fBnumpy\fR
.br
Extensions: \fB.npy\fR, \fB.npz\fR
.br
Read: lazy scan
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
ELF
\-\-format: \fBelf\fR
.br
Extensions: \fB.elf\fR, \fB.axf\fR
.br
Read: in memory
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
ULog
\-\-format: \fBulog\fR
.br
Extensions: \fB.ulg\fR
.br
Read: lazy scan
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
DataFlash
\-\-format: \fBdataflash\fR
.br
Extensions: none: by content
.br
Read: lazy scan
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
candump
\-\-format: \fBcandump\fR
.br
Extensions: none: by content
.br
Read: lazy scan
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
Text
\-\-format: \fBtext\fR
.br
Extensions: \fB.log\fR, \fB.txt\fR
.br
Read: lazy scan
.br
Compressed: decompressed copy
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
systemd journal
\-\-format: \fBjournal\fR
.br
Extensions: none: by content
.br
Read: in memory
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
Arrow IPC stream
\-\-format: \fBarrow\fR
.br
Extensions: as Arrow IPC
.br
Read: converted to Arrow
.br
Compressed: no
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: downloaded
.TP
Format spec
\-\-format: its name
.br
Extensions: its \fBmatch\fR
.br
Read: lazy scan
.br
Compressed: decompressed copy
.br
HTTP(S): downloaded
.br
In a bucket: downloaded
.br
Bucket prefix: no
.TP
lazy scan
Scanned where it is. Browsing reads a buffer of rows; queries, sorting and analysis may read the whole input
.TP
decompressed copy
Decompressed whole into a temporary file of the same format in the temp directory (\fB\-\-temp\-dir\fR), which is then scanned lazily. The file is removed on quit (temporary files)
.TP
converted to Arrow
Read through whole into a temporary Arrow IPC file in the temp directory, which is then scanned lazily. Removed on quit, like a decompressed copy; nothing is cached between sessions
.TP
in memory
Read whole into memory before the table appears. Past \fBmemory_warning\fR in \fB[read]\fR (\fB"1GiB"\fR by default; 0 never asks), datui asks first: \fBbig.json: JSON reads 2.0 GiB into memory\fR. A model file's table is one row per tensor, from the header, so it is small however large the model, and is never asked about; a MIDI file is at most 64 MiB (\fBmidi_bytes\fR in \fB[limits]\fR)
.IP \(bu 4
\fBCompressed\fR is a \fB.gz\fR, \fB.zst\fR, \fB.bz2\fR or \fB.xz\fR file; \fBno\fR means it does not open. \fB\-c read.\:decompress_in_memory=true\fR reads compressed CSV, TSV, PSV and text in memory instead.
.IP \(bu 4
\fBHTTP(S)\fR is one file at an \fBhttp://\fR or \fBhttps://\fR URL. \fBdownloaded\fR copies it to the temp directory first, then reads it as \fBRead\fR says. A model file's header is fetched by range; a server that sends no ranges gets the download question.
.IP \(bu 4
\fBIn a bucket\fR is one S3, GCS or Azure object. \fBin place\fR reads only what is needed with ranged requests: a Parquet or Arrow IPC file's footer and the rows shown, or a model file's header. \fBdownloaded\fR copies the object to the temp directory first, after asking, then reads it as \fBRead\fR says; an Arrow stream is converted as it downloads, with no copy of the stream kept.
.IP \(bu 4
\fBBucket prefix\fR is a prefix or glob read as one table. Only Parquet reads hive partitions; the model files directly under a prefix are read by their headers. An Arrow prefix scans its IPC files in place and downloads its streams, one split of a Hugging Face cache as on disk; a glob of Arrow reads IPC files only. A prefix marked \fBno\fR opens one object at a time from the cloud source.
.IP \(bu 4
Standard input is written to a temporary file first, then read as \fBRead\fR says.
.PP
The home screen marks a file row that is not read lazily where it is: \fBdecompresses\fR, \fBconverts\fR, \fBin memory\fR or \fBdownloads\fR. The details pane and the Info panel's Resources tab say how it is read.
.SH "DETECTED BY CONTENT"
The format comes from the first bytes, unless \fB\-\-format\fR or \fB\-\-compression\fR names it:
.TP
Parquet, Arrow IPC or Avro magic number, or an Arrow IPC stream's schema message
that format
.TP
\fBSQLite format 3\fR
SQLite; a database of several tables needs \fB\-\-table\fR
.TP
gzip, zstd, bzip2 or xz magic number
decompressed, then read by what is inside, as below: a text format, CSV or TSV, or lines
.TP
\fB{\fR, the first line an object with \fB__CURSOR\fR and \fB__REALTIME_TIMESTAMP\fR
systemd journal
.TP
\fB[\fR, then JSON
JSON
.TP
\fB{\fR, the first line a whole object
NDJSON
.TP
\fB{\fR, the object open past the first line, JSON so far
JSON
.TP
an NMEA sentence (\fB$GPGGA,\fR) with a checksum that matches, or of a type receivers write
NMEA
.TP
XML whose first element is \fB<gpx\fR
GPX
.TP
a line with \fB8=FIX\fR, a delimiter and \fB9=\fR
FIX
.TP
\fB$date\fR, \fB$version\fR, \fB$timescale\fR, \fB$comment\fR, \fB$scope\fR or \fB$var\fR, with an \fB$end\fR
VCD
.TP
a \fBV2000\fR or \fBV3000\fR counts line, or \fBM END\fR with a data item or \fB$$$$\fR
SDF
.TP
several lines with one field count, two or more, split at tabs
TSV
.TP
several lines with one field count, two or more, split at commas, quotes where CSV allows them
CSV
.TP
anything else
lines
.PP
A comma or a tab alone is not a table: a log line with a comma in it stays a line. An unnamed CSV the first lines do not show as one opens with \fB\-\-format csv\fR; the Info panel's notes say so. With \fB\-\-format csv\fR, \fBtsv\fR or \fBpsv\fR and no \fB\-\-compression\fR, compression still comes from the first bytes.
.SH "FORMAT SPECS"
A format spec is a TOML file that describes a binary format, or a family of delimited text files, so datui opens it as a table.
.PP
\fB\fBl2feed.toml\fR\fR
.PP
.RS 4
.EX
name = "acme.l2feed"
description = "Level 2 capture"
match = { glob = ["*.l2"], magic = "L2FD" }
endian = "le"
[header]
fields = [{ name = "magic", type = "str", size = 4 }, { name = "count", type = "u1" }]
[records]
count = "header.count"
fields = [
{ name = "ts", type = "u8", time = "ns" },
{ name = "symbol", type = "str", size = 8 },
{ name = "side", type = "u1", enum = { 1 = "BUY", 2 = "SELL" } },
{ name = "price", type = "u4", scale = 4, null = "max" },
]
.EE
.RE
.PP
\fB\fBmake_day_l2.py\fR\fR writes \fBday.l2\fR, a file in that format:
.PP
.RS 4
.EX
import ctypes
class Header(ctypes.LittleEndianStructure):
_layout_ = "ms"
_pack_ = 1 # no padding between fields
_fields_ = [
("magic", ctypes.c_char * 4),
("count", ctypes.c_uint8),
]
class Record(ctypes.LittleEndianStructure):
_layout_ = "ms"
_pack_ = 1
_fields_ = [
("ts", ctypes.c_uint64), # nanoseconds since 1970
("symbol", ctypes.c_char * 8),
("side", ctypes.c_uint8), # 1 BUY, 2 SELL
("price", ctypes.c_uint32), # in ten\-thousandths; the largest value is null
]
records = [
Record(1709294400000000000, b"MSFT", 1, 4105000),
Record(1709294400000500000, b"AAPL", 2, 0xFFFFFFFF),
]
with open("day.l2", "wb") as f:
f.write(bytes(Header(b"L2FD", len(records))))
for record in records:
f.write(bytes(record))
.EE
.RE
.PP
.RS 4
.EX
python3 make_day_l2.py
datui formats check ./l2feed.toml day.l2
datui \-\-format ./l2feed.toml day.l2
mkdir \-p formats
cp l2feed.toml formats/
DATUI_FORMATS_PATH=formats datui day.l2
DATUI_FORMATS_PATH=formats datui formats
.EE
.RE
.TP
\fBdatui formats check ./l2feed.toml day.l2\fR
Checks the spec and prints the file's header and first rows
.TP
\fBdatui \-\-format ./l2feed.toml day.l2\fR
Reads the file with that spec file
.TP
\fBdatui \-\-format acme.l2feed day.l2\fR
Reads it with the spec of that name, from the search path
.TP
\fBdatui day.l2\fR
Reads it with the spec whose \fBmatch\fR it fits, from the search path
.TP
\fBdatui formats\fR
Lists the specs and dictionaries on the search path
.PP
A spec reads fixed-size records, records that carry their length, several message types in one stream, or compressed blocks; a \fBkind = "delimited"\fR spec reads CSV-like text with lines above its header. Specs are data: no scripts or expressions, and every size read from a file is bounded. \fB\-\-format\fR also takes a spec's \fBhttp(s)://\fR, \fBs3://\fR, \fBgs://\fR or \fBaz://\fR URL, fetched once as the open starts; a spec file is at most 1 MiB.
.SS "A spec"
\fBl2feed.toml\fR above is a whole spec. Its table has the columns \fBts\fR (a datetime), \fBsymbol\fR, \fBside\fR and \fBprice\fR (a decimal with four places, null where the field holds its largest value), one row per record. Format spec reference lists every field type and key.
.TP
\fBname\fR
The format's name, namespaced: \fBacme.l2feed\fR. \fB\-\-format\fR takes it
.TP
\fBdescription\fR
Shown by \fBdatui formats\fR and the Documentation view
.TP
\fBdocumentation\fR
An \fBhttps://\fR link to the format's own documentation, for the Documentation view
.TP
\fBmatch\fR
Which files are this format: \fBglob\fR (a pattern or a list), \fBmagic\fR (a string or a list of bytes) at \fBmagic_offset\fR (default 0), \fBwhere\fR (header values)
.TP
\fBendian\fR
\fBle\fR (default), \fBbe\fR, or \fBauto\fR: big-endian when the magic (at least two bytes) reads reversed. For fields without their own suffix
.TP
\fBlayout\fR
\fBrows\fR (default): one file of records. \fBcolumns\fR: a directory with one file per field
.TP
\fB[header] fields\fR
Fields read once from the start of the file. Later parts refer to them
.TP
\fB[header] size\fR
The header's size when it is more than its fields; a number or a header field
.TP
\fB[records] fields\fR
The fields of one record, in order
.TP
\fB[records] size\fR
The record's size, at least the fields' sum (the default); the rest is skipped
.TP
\fB[records] count\fR
How many records there are, such as \fB"header.count"\fR or \fB"footer.n"\fR
.TP
\fB[records] framing\fR
\fBfixed\fR (default), \fBlength_prefixed\fR, \fBvariant\fR or \fBsync\fR: see records of different sizes
.TP
\fB[records] ring\fR
A ring buffer of fixed records: the oldest record's index, such as \fB"header.write_idx"\fR; rows start there and wrap
.TP
\fB[records] checksum\fR
A checksum in each record: see checks
.TP
\fB[[variants]]\fR
Record layouts a type field picks: see variants
.TP
\fB[footer]\fR
Fields at the end of the file: see footer
.TP
\fB[blocks]\fR
Records in blocks, each compressed on its own: see blocks
.TP
\fB[sections.NAME]\fR
A part of the file, \fBoffset\fR and \fBsize\fR from the header, that \fBstring_at\fR fields point into
.TP
\fB[capture]\fR
Records in the UDP payloads of a pcap or pcapng capture: see captures
.TP
\fB[files]\fR
A directory tree of the format's files as one table: see a tree of files
.SS "Where specs live"
Datui reads every \fB*.toml\fR in these places, in order. The first spec of each name wins, as with \fBPATH\fR.
.TP
\fB~/.config/datui/formats/\fR
Your own specs
.TP
\fB$DATUI_FORMATS_PATH\fR
Directories separated by \fB:\fR (\fB;\fR on Windows), such as a checked-out repository of a team's specs
.TP
\fB[formats] path\fR in config
More directories. Lists add up across imported config files
.PP
\fBdatui formats\fR lists each spec, what it matches (\fBmagic L2FD \(pc version 3 \(pc glob *.l2\fR, or \fBno match\fR), the file it came from, any copy of the same name it overrides, and the files that could not be read, with the line and column of each problem. The same places hold FIX log dictionaries: QuickFIX XML files and TOML files of \fBkind = "fix"\fR, listed after the specs.
.PP
\fBdatui formats check SPEC [FILE]\fR checks one spec, by name or file. With a file, it prints the warnings and the first ten rows. Given a QuickFIX dictionary, it checks that, and with a FIX log says how many messages it matches and which of its tags they hold. It exits non-zero on an error, so a repository of specs can run it in CI.
.SS "Which spec reads a file"
.TP
\fB\-\-format FILE\fR: a path (it has a \fB/\fR or ends \fB.toml\fR), or an \fBhttp(s)://\fR, \fBs3://\fR, \fBgs://\fR or \fBaz://\fR URL fetched once as the open starts
That spec, whatever the file is called
.TP
\fB\-\-format NAME\fR
The spec of that name
.TP
A name datui already reads (\fB.csv\fR, \fB.parquet\fR)
Read as it is, as before, unless a delimited spec matches a \fB.csv\fR, \fB.tsv\fR or \fB.psv\fR
.TP
A \fBglob\fR matches
That spec
.TP
A \fBmagic\fR matches, in a file whose bytes are no format datui reads (such as Parquet)
That spec
.PP
A spec with \fBmatch.where\fR matches only a file whose header holds those values, so one spec per version can share a glob and a magic. In a spec, in place of its \fBmatch\fR line:
.PP
.RS 4
.EX
match = { glob = "*.l2", magic = "L2FD", where = { "header.version" = 3 } }
.EE
.RE
.PP
When two specs match the same way, the first on the search path reads the file. The bar shows \fB2 formats match\fR, and the Notes tab names the others. A file no spec matches opens as it does without specs; a local file no reader takes either opens in the hex view, where \fBB\fR reads it with a spec and \fBr\fR lines the bytes up in records while you write one.
.PP
\fBT\fR on a file of several record types lists the whole file, then each type with its column count, and opens the one picked (\fBday.itch/add\fR), clearing the query, filters and sort.
.PP
\fBb\fR on the table picks another spec and reads the file again with it, clearing the query, filters and sort. The list starts with the spec the file was read with, then the others that matched it the same way, then every other spec on the search path for a file (or for a directory of column files). The Notes tab of \fBi\fR says which spec read the file and why (\fBmatched by magic L2FD \(pc version 3\fR), the header's values, and any bytes left out.
.PP
The home screen and its search name a file the same way. A file whose name says nothing (no extension, or \fB.bin\fR) is matched by magic and \fBwhere\fR against the first 4 KiB the listing reads from it anyway, up to 256 files a directory; nothing more is read. Its row reads the spec's name, and its details:
.TP
\fBkind\fR
\fBacme.l2feed file\fR
.TP
\fBspec\fR
The spec's file, cut in the middle to fit: \fB~/\[u2026]/formats/l2feed.toml\fR
.TP
\fBmatch\fR
What named the file, a chip per condition: \fB[magic L2FD] [version 3]\fR when the magic and the header did, \fB[glob *.l2]\fR when the glob did
.TP
\fBschema\fR
\fB3 columns (spec)\fR and each column's type, when the spec alone says them (fixed records, no size from the header); otherwise \fBon open\fR
.TP
\fBrecords\fR
For a spec with variants: \fB2 types (spec)\fR and each record type's column count (\fBadd 5 \(pc cancel 3\fR). \(-> lists the record types
.PP
A chip with several values (\fB[glob *.l2 *.lvl2]\fR) takes any of them. The listing names a file by its glob without reading it, so a glob-named row shows no \fBwhere\fR values; the open still checks them. Chips are drawn without brackets where the header tint shows. Text values are quoted where it does not (\fB[kind "A"]\fR), and a magic that is not text is hex (\fB7f 45 4c 46\fR).
.SS "Delimited text"
Loggers and instruments write a metadata line and a units line above a padded header. A spec of \fBkind = "delimited"\fR holds the CSV options for such a family of files, so they open with no flags: from the command line, from the home screen, compressed, or as a directory.
.PP
\fB\fBinstrument.toml\fR\fR
.PP
.RS 4
.EX
name = "acme.instrument\-log"
kind = "delimited"
match = { magic = "#device_info" }
comment = "#"
skip_initial_space = true
header_rows = { name = 3, unit = 2 }
metadata_line = 1
[columns]
time = { from = ["Lcl Date", "Lcl Time", "UTCOfst"], as = "datetime" }
bus1volts = { description = "Main bus voltage" }
.EE
.RE
.PP
\fB\fBflight.csv\fR\fR
.PP
.RS 4
.EX
#device_info, log_version="1.03", model="Unit 7, rev B", serial="123"
#yyyy\-mm\-dd, hh:mm:ss, hh:mm, degrees, volts, deg F
Lcl Date, Lcl Time, UTCOfst, Latitude, bus1volts, T1 Temp
, , , , 25.1, 187.2
2024\-03\-01, 10:00:00, \-05:00, 40.100000, 25.0, 180.0
.EE
.RE
.PP
.RS 4
.EX
datui formats check ./instrument.toml flight.csv
datui \-\-format ./instrument.toml flight.csv
.EE
.RE
.PP
A delimited spec takes the keys of the config's [\fB[csv]\fR](.\:.\:/\:reference/\:settings.\:md#csv), plus \fBmatch\fR, \fBkind\fR, the layout keys, \fB[columns]\fR, \fBdescription\fR and \fBdocumentation\fR.
.TP
\fBkind\fR
\fBdelimited\fR. Default \fBbinary\fR
.TP
\fBmatch\fR
\fBglob\fR and \fBmagic\fR, as for other format specs. \fBmagic\fR compares the start of the first line
.TP
\fBdelimiter\fR
One character, \fB"tab"\fR, \fB"\et"\fR or a code such as \fB"0x1f"\fR, as \fB\-\-delimiter\fR takes. Default \fB,\fR, or the one the file's name implies
.TP
\fBcomment\fR
Lines that start with it are skipped wherever they are
.TP
\fBskip_initial_space\fR
\fBtrue\fR: ignore the spaces after a delimiter
.TP
\fBheader_rows\fR
\fB{ name = N, unit = M }\fR: the line that names the columns and the line that gives their units. \fBname\fR may be a list of lines, joined with \fBheader_join\fR (default a space). A number or a list is \fBname\fR alone. A header line is never data
.TP
\fBheader_join\fR
What joins the pieces of a name from several lines
.TP
\fBmetadata_line\fR
A line of \fBkey="value"\fR or \fBkey=value\fR pairs, separated by commas, for the Info panel. It must not be data: above the last header line, within \fBskip_lines\fR, or a comment line
.TP
\fBnull_values\fR
A value, or a list, read as null: \fB"NA"\fR, or \fB"COL=\-999"\fR for one column
.TP
\fBskip_lines\fR
Lines to pass over before the header
.TP
\fB[columns]\fR
Column types and derived columns, below, and what columns mean: \fBdescription\fR and \fBunit\fR
.PP
Lines count from 1 at the top of the file. Each option the spec sets replaces the config's; a flag typed on the command line (\fB\-\-delimiter\fR, \fB\-\-comment\fR, \fB\-\-skip\-initial\-space\fR, \fB\-\-header\-rows\fR, \fB\-\-skip\-lines\fR) wins over the spec. The options the spec does not set keep theirs. \fBdatui \-\-delimiter ';' formats check SPEC FILE\fR reads the file as an open with those flags would, and names the flags that override the spec. The header lines and the metadata line are the only lines read apart from the CSV reader.
.PP
\fBUnits\fR
.PP
A unit sits beside its column's type on the table's type row (\fBf64 \(pc deg F\fR), in a \fBUnit\fR column on the Info panel's Schema tab, and in chart axis titles (\fBT1 Temp (deg F)\fR). A filter, sort or drill keeps them, and so does a query, pivot or melt for each column it carries unchanged, renamed or not. A column a query computes has no unit, even under the name of one that had.
.PP
\fBMetadata\fR
.PP
The Info panel's \fBMetadata\fR tab lists the metadata line's pairs, under its leading item when it has one (\fBdevice_info\fR). A line that is not pairs is shown as it is. For a directory, the first file's line is shown.
.PP
\fBColumn types\fR
.PP
A column of the file takes a \fBtype\fR, beside its \fBunit\fR and \fBdescription\fR:
.PP
\fB\fBtyped.toml\fR\fR
.PP
.RS 4
.EX
name = "acme.typed\-log"
kind = "delimited"
match = { magic = "#device_info" }
comment = "#"
skip_initial_space = true
header_rows = { name = 3, unit = 2 }
metadata_line = 1
[columns]
"Lcl Date" = { type = "date", format = "%Y\-%m\-%d" }
Latitude = { type = "f64", description = "GPS latitude" }
bus1volts = { type = "f64", unit = "V" }
.EE
.RE
.PP
\fB\fBtyped.csv\fR\fR
.PP
.RS 4
.EX
#device_info, log_version="1.03"
#yyyy\-mm\-dd, degrees, volts
Lcl Date, Latitude, bus1volts
2024\-03\-01, 40.100000, 25.0
2024\-03\-01, , n/a
.EE
.RE
.PP
.RS 4
.EX
datui formats check ./typed.toml typed.csv
.EE
.RE
.TP
\fBstr\fR
Text as it is, never typed by \fBread.infer_types\fR: \fB02134\fR keeps its zero
.TP
\fBbool\fR
\fBtrue\fR/\fBfalse\fR or \fB1\fR/\fB0\fR, in any case
.TP
\fBi8\fR \fBi16\fR \fBi32\fR \fBi64\fR
Signed integers
.TP
\fBu8\fR \fBu16\fR \fBu32\fR \fBu64\fR
Unsigned integers
.TP
\fBf32\fR \fBf64\fR
Decimals. \fBf32\fR keeps about 7 significant digits
.TP
\fBdate\fR \fBtime\fR \fBdatetime\fR
With \fBformat\fR, a strftime format; without, the format is inferred
.TP
\fBduration\fR
\fB1d\fR, \fB2h30m\fR, \fB\-1w2d\fR
.PP
Use \fBi64\fR and \fBf64\fR unless a narrower type is wanted for an export or to hold values to a range. A value is trimmed first, and one that does not fit the type, or is out of an integer type's range, is null. The first time the Info panel opens, one pass counts them, and the Notes tab says how many per column: \fBRPM: 2 values out of range for u8, read as null\fR. A typed column the file does not have is a note, not an error, since the files of a family differ. A typed column is the same type in every file read together, and \fBread.infer_types\fR leaves it alone. \fBtype\fR beside \fBfrom\fR or \fBas\fR is refused: a derived column takes its type from \fBas\fR. The same types, and the derived columns, are on hand in the table: Column types.
.PP
\fBDerived columns\fR
.TP
\fBdatetime\fR
from: a date and a time, and optionally a UTC offset such as \fB\-05:00\fR, \fB+0530\fR or \fB\-5\fR; or one column of text
.br
Column: A datetime. With an offset it is in UTC
.TP
\fBdate\fR
from: one column
.br
Column: A date
.TP
\fBtime\fR
from: one column
.br
Column: A time of day
.PP
A column of the file takes \fBdescription\fR and \fBunit\fR alone (\fBbus1volts = { description = "Main bus voltage" }\fR), and a derived one takes them beside \fBfrom\fR and \fBas\fR. They show in the Documentation view. A \fBunit\fR there is documentation only: the type row shows the units line's.
.PP
\fBformat = "%Y\-%m\-%d %H:%M:%S"\fR gives the strftime format of the text, a date and a time joined with a space; without it the format is inferred. A value that does not parse is null. The column goes before the first column it is made from, which stays; one named after a column it is made from replaces that column, and its unit. There is no expression language: anything more is a query.
.PP
\fBMatching\fR
.PP
A delimited spec matches a file whose name says no format datui reads, or says \fB.csv\fR, \fB.tsv\fR or \fB.psv\fR, compressed or not. A directory, or a glob such as \fB'logs/log_*.csv'\fR, is read through the spec its first file with text matches. \fBH\fR on the Info panel's Schema tab reads the file without a header, and without its derived columns.
.PP
\fBSeveral files\fR
.PP
Files read together through a spec are matched by column name, so logs from different writer versions stack:
.TP
A file lacks a column
The column is null in its rows
.TP
A column is blank in the first rows a file's types are inferred from
It takes the type the other files give it. A value further on that is not of that type stops the read, naming the file and the column
.TP
One file's column holds integers and another's decimals
The column is \fBf64\fR
.TP
A file's column holds text where another's holds numbers
The column is text
.TP
Files give a column different units
The first file's unit; a note lists the units seen
.PP
The Notes tab lists the columns not every file has. Columns keep the order the files first have them in.
.SS "Garmin TXi logs"
Garmin and TXi are trademarks of Garmin Ltd. or its subsidiaries; datui is not affiliated with or endorsed by Garmin.
.PP
The repository's \fBcontrib/\:formats/\:garmin\-txi.\:toml\fR reads the data logs a Garmin TXi writes: the airframe line as metadata, the units line, \fBtime\fR in UTC, and each column typed. Copy it into \fB~/.config/datui/formats/\fR to open the logs, or a directory of them, with no flags. A twin fills the \fBE2\fR columns and a single leaves them blank. A log written before a GPS fix has blank date and GPS cells.
.PP
\fB\fBgarmin\-log.csv\fR\fR
.PP
.RS 4
.EX
#airframe_info, log_version="1.03", airframe_name="Example 182", tail_number="N12345", system_id="0000EXAMPLE", unit="GDU1",
#yyy\-mm\-dd, hh:mm:ss, hh:mm, ident, degrees, degrees, ft msl, kt, rpm, deg F, deg F, bool, #
Lcl Date, Lcl Time, UTCOfst, AtvWpt, Latitude, Longitude, AltMSL, IAS, E1 RPM, E1 CHT1, E1 EGT1, OnGrnd, LogIdx
, , , , , , , 0.0, 980.0, 210.0, 1105.0, 1, 1
2024\-05\-04, 09:12:01, \-04:00, KXYZ, 41.0000000, \-74.0000000, 350.0, 0.0, 1000.0, 215.0, 1120.0, 1, 2
2024\-05\-04, 09:12:02, \-04:00, KXYZ, 41.0000100, \-74.0000100, 350.0, 12.5, 1800.0, 230.0, 1250.0, 0, 3
.EE
.RE
.PP
.RS 4
.EX
datui formats check contrib/formats/garmin\-txi.toml garmin\-log.csv
.EE
.RE
.SS "Checks"
.TP
A field, size or key the spec gets wrong
The spec is refused, with its line and column
.TP
The magic does not match
The open fails, showing the bytes found
.TP
A file that ends partway through a record
The whole records open; a note shows the bytes left over
.TP
A header count larger than the file
The whole records open, with a note
.TP
A record whose length or type cannot be read
The records before it open; a note says where the rest was left out
.TP
\fBchecksum\fR in \fB[records]\fR
A \fBchecksum_ok\fR column, true or false for each record, rather than an error
.TP
A spec for a file in a bucket or at a URL
The file is downloaded first, then read
.PP
In a spec's \fB[records]\fR:
.PP
.RS 4
.EX
checksum = { algo = "crc16\-ccitt", field = "crc", from = "len", to = "crc" }
.EE
.RE
.PP
A record checksum covers the bytes from the \fBfrom\fR field (default: the record's start) up to the \fBto\fR field (default: the checksum's own field). \fBalgo\fR is \fBcrc16\-ccitt\fR, \fBcrc16\-xmodem\fR, \fBcrc16\-modbus\fR, \fBcrc16\-arc\fR, \fBcrc32\fR, \fBcrc32c\fR, \fBsum8\fR or \fBxor8\fR; a footer checksum takes the same names.
.SS "Compressed files"
\fBday.l2.zst\fR, \fB.gz\fR, \fB.bz2\fR and \fB.xz\fR are decompressed to a temporary file before they are read: the loading screen says \fBDecompressing\fR, then \fBReading records\fR, and Esc stops either. The glob matches the name without the compression suffix, and magic is read from the decompressed bytes.
.SS "Large files"
A spec's file is read as a lazy scan, or from a decompressed copy when compressed. It is memory-mapped, and only the columns and rows on screen are decoded. Records that are not all one size, and blocks, are indexed by one pass when the file opens. That pass keeps where each record starts (5 bytes a record, up to 64M records), so a query reads every column from there rather than walking the records again, and a file opened again with the same spec is not walked again. Scrolling to the last row of a gigabyte file reads only the rows shown. A sort, filter, query, chart or analysis reads every row of the columns it uses, a batch at a time on the streaming engine (\fB[performance] streaming\fR, on by default). A table holds at most 4,294,967,295 rows; records past that are not shown, and the dataset's notes say so.
.PP
A file that grows while it is open keeps the rows it had; open it again to read the rest. A file cut short by another program is refused at the next read rather than read past its end.
.SH "FORMAT SPEC REFERENCE"
Every field type and key a format spec takes. Each example below is a whole spec; \fBdatui formats check ./spec.toml\fR checks one.
.SS "Field types"
Type names follow Kaitai Struct <\fIhttps://kaitai.io\fR>. Widths are in bytes.
.TP
\fBu1\fR to \fBu8\fR, \fBs1\fR to \fBs8\fR
Unsigned and signed integers of 1 to 8 bytes, \fBu3\fR and \fBs6\fR included
.TP
\fBf2\fR, \fBf4\fR, \fBf8\fR
Floats; \fBf2\fR is a half float
.TP
\fBbf2\fR
A bfloat16
.TP
\fBvu\fR, \fBvs\fR
LEB128 varints, \fBvs\fR zigzag-encoded. Records only
.TP
\fBbool\fR
One byte, nonzero is true
.TP
\fBstr\fR
Text of \fBsize\fR bytes, its NUL and space padding trimmed
.TP
\fBstrz\fR
Text up to a NUL, at most \fBsize\fR bytes when given. Records only
.TP
\fBbytes\fR
Raw bytes of \fBsize\fR
.TP
\fBpad\fR
\fBsize\fR bytes skipped, no column
.PP
A \fBle\fR or \fBbe\fR suffix (\fBu4be\fR, \fBs2le\fR, \fBbf2be\fR) overrides the spec's \fBendian\fR.
.SS "Field keys"
.TP
\fBname\fR
The column's name. Every field but \fBpad\fR has one
.br
Example: \fB"price"\fR
.TP
\fBsize\fR
Bytes of a \fBstr\fR, \fBstrz\fR, \fBbytes\fR or \fBpad\fR: a number, a header or footer field, an earlier field of the record, or \fBrest\fR, what is left of the record
.br
Example: \fB8\fR, \fB"header.len"\fR, \fB"len"\fR, \fB"rest"\fR
.TP
\fBsize_adjust\fR
Added to a size read from a field
.br
Example: \fB\-4\fR
.TP
\fBencoding\fR
Of a \fBstr\fR or \fBstrz\fR: \fButf8\fR (default), \fBlatin1\fR, \fButf16le\fR or \fButf16be\fR
.br
Example: \fB"latin1"\fR
.TP
\fBcount\fR
That many values side by side: one Array column
.br
Example: \fB10\fR
.TP
\fBflatten\fR
With \fBcount\fR (up to 1024), columns \fBname_0\fR to \fBname_9\fR instead of an Array
.br
Example: \fBtrue\fR
.TP
\fBnull\fR
A stored value that means no value: the type's smallest or largest value, a NaN, or this number, which the type must be able to hold
.br
Example: \fB"min"\fR, \fB"max"\fR, \fB"nan"\fR, \fB\-1\fR
.TP
\fBscale\fR
Implied decimal places: the integer becomes a Decimal
.br
Example: \fB4\fR
.TP
\fBfactor\fR, \fBoffset\fR
\fBvalue * factor + offset\fR, as a float
.br
Example: \fB0.1\fR, \fB\-40.0\fR
.TP
\fBenum\fR
Codes and labels; an unlisted code reads as its number
.br
Example: \fB{ 1 = "BUY", 2 = "SELL" }\fR
.TP
\fBtime\fR
A count of \fBdays\fR, \fBs\fR, \fBms\fR, \fBus\fR or \fBns\fR: a datetime, or a date for \fBdays\fR. A float counts fractions too
.br
Example: \fB"ns"\fR
.TP
\fBepoch\fR
What the count is since. Default 1970-01-01
.br
Example: \fB2000\-01\-01\fR
.TP
\fBdate\fR
An integer such as 20240102, as a date
.br
Example: \fB"yyyymmdd"\fR
.TP
\fBof_day\fR
With \fBtime\fR, a count since midnight: a time of day
.br
Example: \fBtrue\fR
.TP
\fBdate\fR with \fBof_day\fR
The day those times are on, from a header field that reads as a date (\fBdate = "yyyymmdd"\fR, \fBtime = "days"\fR), a datetime, or text such as \fB2024\-01\-02\fR: a datetime
.br
Example: \fB"header.trade_date"\fR
.TP
\fBfile\fR
In the columns layout, the file in the directory holding the field. Default: its name
.br
Example: \fB"px.dat"\fR
.TP
\fBoffset\fR
In the columns layout, where the column starts in one file: see columns layout
.br
Example: \fB"header.px_off"\fR
.TP
\fBlookup\fR
An integer indexes a list of symbols in a file beside the data: a categorical. \fBformat\fR is \fBlines\fR (default), \fBnul\fR or \fBstr:N\fR
.br
Example: \fB{ file = "../sym", format = "lines" }\fR
.TP
\fBdescription\fR
What the column means, for the Documentation view. Not read
.br
Example: \fB"Limit price"\fR
.TP
\fBunit\fR
The column's unit, for the Documentation view. Not read
.br
Example: \fB"USD"\fR
.PP
These keys are for record fields only:
.TP
\fBdelta\fR
Each value is the change from the record before; the running sum is shown. \fB"block"\fR starts the sum again in each block
.br
Example: \fBtrue\fR, \fB"block"\fR
.TP
\fBbits\fR
Bit fields of an integer, each its own column. A width of 1 is a bool
.br
Example: \fB[{ name = "valid", bit = 0 }, { name = "mode", bit = 4, width = 3, enum = { 0 = "IDLE" } }]\fR
.TP
\fBgroup\fR
A counted run of items: a List of Structs. Takes no \fBtype\fR
.br
Example: \fB{ count = "n_levels", fields = [...] }\fR
.TP
\fBstring_at\fR
An unsigned offset into a \fB[sections.strings]\fR part of the file, where NUL-terminated text is
.br
Example: \fB"strings"\fR
.PP
A field takes at most one of \fBtime\fR (or \fBdate\fR), \fBscale\fR, \fBfactor\fR and \fBenum\fR.
.PP
A field refers to an earlier one by name, never by an expression. In the header, \fBsize = "len"\fR reads an earlier header field; anywhere, \fBheader.NAME\fR and \fBfooter.NAME\fR do. In a record, \fB"len"\fR reads an earlier field of the same record. A record whose own field gives its size is \fBlength_prefixed\fR.
.SS "Records of different sizes"
.RS 4
.EX
name = "acme.messages"
match = { glob = "*.msg" }
[records]
framing = "length_prefixed"
size = "len" # the field that holds each record's length
size_adjust = 2 # the length leaves out its own two bytes
fields = [{ name = "len", type = "u2" }, { name = "msg", type = "str", size = "rest" }]
.EE
.RE
.TP
\fBfixed\fR
Every record takes \fBsize\fR, or what its fields take
.TP
\fBlength_prefixed\fR
A field of the record gives its size: \fBsize = "len"\fR, with \fBsize_adjust\fR
.TP
\fBvariant\fR
The variant the type field picks gives the size: its \fBsize\fR, or what its fields take
.TP
\fBsync\fR
Each record starts with the \fBsync\fR marker (\fB"1ACFFC1D"\fR, \fB"0xEB90"\fR or a list of bytes); bytes between records are skipped and counted in a note
.TP
\fBlength_suffix\fR
\fBtrue\fR: the length is written again after the record, as Fortran unformatted files do. Needs \fBsize = "len"\fR
.TP
\fBalign\fR
Each record starts at a multiple of this (1 to 65536), counted from the first: \fBalign = 2\fR for IFF and RIFF chunks
.PP
Records that are not all one size are walked once when the file opens, and the start of every 1024th is kept, so a scroll anywhere reads from the nearest one.
.PP
\fBVariants\fR
.PP
.RS 4
.EX
name = "acme.orders"
match = { glob = "*.ord" }
[records]
framing = "length_prefixed"
size = "len"
size_adjust = 2
fields = [{ name = "len", type = "u2" }, { name = "kind", type = "str", size = 1 }]
type = "kind"
[[variants]]
name = "add"
when = "A"
fields = [{ name = "ref", type = "u8" }, { name = "shares", type = "u4" }, { name = "price", type = "u4", scale = 4 }]
[[variants]]
name = "exec"
when = ["E", "C"]
fields = [{ name = "ref", type = "u8" }, { name = "shares", type = "u4" }]
.EE
.RE
.PP
\fBfields\fR (or \fB[records.common] fields\fR) are the fields every record starts with. \fBtype\fR names the common field that picks the variant: an integer, or text, compared with its padding trimmed. \fBtype = { field = "kind", type = "u1" }\fR declares it in place.
.TP
\fBname\fR
Shown in the \fBtype\fR column
.TP
\fBwhen\fR
The type value, or a list of them, that picks it
.TP
\fBfields\fR
The fields after the common ones
.TP
\fBsize\fR, \fBsize_adjust\fR
The whole record's size, when more than its fields take
.TP
\fBdescription\fR
What a record of this type is, for the Documentation view
.PP
All records make one table, with a \fBtype\fR column naming each one's variant. A column of a field one variant lacks is null in that variant's rows. A record of a type no variant names shows as \fB?X\fR when its size is known (\fBlength_prefixed\fR); otherwise the read stops there, with a note.
.PP
A field name two variants share is one column, so it must be the same field in both: the same type, count, encoding, bits and group, and the same \fBtime\fR, \fBscale\fR, \fBfactor\fR or \fBenum\fR. Otherwise the spec is refused: \fBvariants: `px` is a different field in two variants; one column has one type, so name them apart\fR. A variant's field cannot take a common field's name either (\fBa second field named `kind`\fR), nor be named \fBtype\fR.
.PP
\fBdatui \-\-table add day.ord\fR opens one variant as its own table: only its records and its columns.
.PP
On the home screen, each variant of a file a spec reads is a record type: its details read \fBrecords 2 types (spec)\fR and each type's column count (\fBadd 5 \(pc exec 4\fR). Enter opens every record; \(-> lists the record types, one row each at \fBday.ord/add\fR, and Enter on one opens it alone. That path opens the record type on the command line too, and is what recents record.
.SS "Documentation"
A spec says what its files mean with the words a catalog uses. \fBCtrl\fR+\fBE\fR on a file the spec reads, and the Info panel's Documentation tab once it is open, show it in the Documentation view. None of it changes how a file is read.
.PP
.RS 4
.EX
name = "acme.quotes"
description = "Quotes and trades from the Acme feed"
documentation = "https://example.com/acme\-feed.pdf"
match = { glob = "*.acq" }
[records]
framing = "variant"
type = "kind"
fields = [{ name = "kind", type = "u1" }, { name = "ts", type = "u8", time = "ns", description = "When the exchange sent it" }]
[[variants]]
name = "quote"
when = 1
description = "The best bid and offer"
fields = [{ name = "bid", type = "u4", scale = 4, unit = "USD" }, { name = "ask", type = "u4", scale = 4, unit = "USD" }]
[[variants]]
name = "trade"
when = 2
description = "A trade on the book"
fields = [
{ name = "px", type = "u4", scale = 4, description = "Trade price", unit = "USD" },
{ name = "side", type = "u1", enum = { 1 = "BUY", 2 = "SELL" }, description = "The aggressor's side" },
]
.EE
.RE
.TP
\fBdescription\fR
Where: The spec, a variant, a field
.br
Says: What the format, the record type, the column or the header or footer field is
.TP
\fBdocumentation\fR
Where: The spec
.br
Says: An \fBhttps://\fR link to the format's own documentation
.TP
\fBunit\fR
Where: A field
.br
Says: The column's, or the header or footer field's, unit
.TP
\fBenum\fR
Where: A record field
.br
Says: Its codes and labels are the column's value legend
.PP
A flattened field's note goes to each of its columns, \fBbid_0\fR, \fBbid_1\fR and on. Named \fB[header]\fR and \fB[footer]\fR fields with a \fBdescription\fR or \fBunit\fR are listed in sections of their own.
.PP
A delimited spec takes \fBdescription\fR and \fBunit\fR in \fB[columns]\fR, for a column of the file or a derived one: \fBtemp = { description = "Air temperature", unit = "deg F" }\fR. A column's declared \fBtype\fR shows beside its unit: \fBLatitude f64 \(pc deg GPS latitude\fR.
.PP
Each text is trimmed, and an empty one is refused. Where a catalog lists the same file, its description and its \fBdocumentation\fR link stand over the spec's. Where both note a column, each of the catalog's \fBdescription\fR, \fBunit\fR and \fBvalues\fR stands over the spec's when the catalog gives it, and the spec's fills the rest. The spec's other notes and its record types stay.
.SS "Footer"
.RS 4
.EX
name = "acme.counted"
match = { glob = "*.cnt" }
[records]
count = "footer.n"
fields = [{ name = "v", type = "u2" }]
[footer]
fields = [{ name = "n", type = "u4" }, { name = "crc", type = "u4" }]
checksum = { algo = "crc32", field = "crc" }
.EE
.RE
.PP
The footer is read from the end of the file, so its fields' sizes are written in the spec, or it gives \fBsize\fR. Later parts refer to its fields as \fBfooter.NAME\fR: a record count, or a block index's offset. \fBchecksum\fR checks the bytes before the footer against a footer field; a mismatch is a note.
.SS "Blocks"
.RS 4
.EX
name = "acme.blocks"
match = { glob = "*.blk" }
[blocks]
header = [{ name = "clen", type = "u4" }, { name = "rawlen", type = "u4" }]
size = "clen"
compression = "zstd"
uncompressed = "rawlen"
[records]
fields = [{ name = "v", type = "u4", delta = "block" }]
.EE
.RE
.PP
The data after the file's header is a run of blocks: a block header, then \fBsize\fR bytes of records. The records' framing applies inside each block.
.TP
\fBheader\fR
The fields at the start of each block
.TP
\fBsize\fR, \fBsize_adjust\fR
The bytes after the block header: a number or a block header field
.TP
\fBcompression\fR
\fBnone\fR (default), \fBgzip\fR, \fBdeflate\fR, \fBzlib\fR, \fBzstd\fR, \fBlz4\fR, \fBlz4_block\fR, \fBsnappy\fR, \fBsnappy_framed\fR, \fBbrotli\fR, \fBbzip2\fR or \fBxz\fR. Or a code in the block header: \fB{ field = "codec", values = { 0 = "none", 1 = "zstd" } }\fR
.TP
\fBuncompressed\fR
The block header field with the decompressed size. Required for \fBlz4_block\fR
.TP
\fBrecords\fR
The block header field counting its records; missing records are null
.TP
\fBindex\fR
\fB{ at = "footer.index_off", count = "footer.n_blocks", fields = [...] }\fR: a block index read instead of walking the blocks. Its entries need an integer \fBoffset\fR, and may give \fBrows\fR
.PP
Only the block headers are read when the file opens. A block is decompressed when its rows are first read, up to 256 MiB each, and the last few are kept. A block that will not decompress is left out, with a note.
.SS "Captures"
.RS 4
.EX
name = "acme.multicast"
match = { glob = "*.pcap" }
[capture]
header = [{ name = "session", type = "str", size = 10 }, { name = "seq", type = "u8" }, { name = "count", type = "u2" }]
count = "count"
time = "captured"
[records]
fields = [{ name = "price", type = "u4" }]
.EE
.RE
.PP
The file is a pcap or pcapng capture, told apart by its magic. Each UDP payload holds the records, after the payload \fBheader\fR; \fBcount\fR names the header field counting them, and \fBtime\fR adds a column with each packet's capture time. Packets that are not UDP are left out and counted in a note. A capture spec has no \fB[header]\fR, \fB[footer]\fR or \fB[blocks]\fR.
.SS "A tree of files"
.RS 4
.EX
name = "acme.trades"
match = { glob = "*.bin" }
[files]
path = "{date:%Y%m%d}/{venue}/trades.bin"
[records]
fields = [{ name = "price", type = "f8" }]
.EE
.RE
.PP
\fBdatui \-\-format acme.trades store/\fR reads every file under \fBstore/\fR that the pattern matches as one table, with a column for each part: a date for a part with a format, text otherwise. A file whose part does not parse as its date is left out, with a note.
.SS "Columns layout"
.RS 4
.EX
name = "kdb.trades"
layout = "columns"
endian = "be"
[records]
fields = [{ name = "price", type = "f8" }, { name = "size", type = "s8" }]
.EE
.RE
.PP
With it, \fBdatui \-\-format kdb.trades db/trades/\fR reads \fBdb/trades/price\fR and \fBdb/trades/size\fR as two columns of one table, as kdb+ splays a table. A \fB[header]\fR describes the start of each file. A glob in a columns spec matches the directory.
.PP
When the header lists where each column starts in one file, every field gives \fBoffset\fR and the spec reads that one file:
.PP
.RS 4
.EX
name = "acme.packed"
layout = "columns"
[header]
fields = [{ name = "n", type = "u4" }, { name = "px_off", type = "u4" }]
[records]
count = "header.n"
fields = [{ name = "px", type = "f8", offset = "header.px_off" }]
.EE
.RE
.SH "SEE ALSO"
\fBdatui\fR(1), \fBdatui\-formats\fR(1)
.PP
The datui documentation: <https:/\:/\:derekwisong.\:github.\:io/\:datui/\:>
.SH BUGS
Report bugs at <https:/\:/\:github.\:com/\:derekwisong/\:datui/\:issues>.
.SH AUTHORS
Derek Wisong and the datui contributors.
.SH COPYRIGHT
Copyright \(co 2026 Derek Wisong
.PP
datui is free software under the MIT License.