Module formats 

Binary formats described by specs: one TOML file per format, found on a search path.

A spec names its format (acme.l2feed), says which files are in it (match: a glob, magic bytes, header values), and lays out its header and its records. Specs are data: no expressions and no code. A field refers to an earlier one by name instead, and every size read from a file is bounded.

What a spec describes is decoded by crate::fixed_records; this module turns a spec and a file into that reader’s columns.

Modules

files

A directory of one spec’s files, {date}/{venue}/trades.bin: one table, the parts of each file’s path as columns.

Structs

Asked

What a request for a format says, besides the path.

BitField

One bit field of an integer: its own column.

BlockIndex

Blocks listed in the file, so their headers are not walked.

Blocks

Records in blocks, each with a header and, often, compressed on its own.

Capture

A packet capture whose payloads hold the records.

Checksum

A checksum over part of each record, or of the file before its footer.

Choice

Where a spec was chosen from, carried to a decompressed copy’s read.

DbcFound

A DBC file found on the search path.

Field

One field of a header or a record.

Files

A directory of one spec’s files: each file’s path is read by pattern.

FixFound

A FIX dictionary found on the search path, and the copies of the same name it hides.

Footer

Fields at the end of the file.

Found

A spec found on the search path, and the copies of the same name it hides.

Header

A spec’s header: its fields, and its size when that is more than they take.

HeaderValues

A header, read: each named field’s value, and how many bytes the header takes; and the footer’s values, and the symbol lists fields index into.

Lookup

Where a symbol list is, for a field of indexes into it.

MatchChip

One condition of a spec’s match, as the home pane, the Info panel and datui formats show it: magic MKTD, version 1, glob *.bin *.dat. Separate chips must all hold, but for a glob and a magic: a file the glob names is not asked for its magic. Alternatives live inside one chip.

Matched

The specs a file matches, by the first rule that matched any.

Opened

A file read through a spec: its columns, and what the read had to say.

PathPart

A part of a file’s path that becomes a column.

Read

A file read through a spec, as the open carries it to the dataset.

RecordType

One variant of a spec, as its documentation shows it.

Records

A spec’s records.

Registry

Every spec on the search path, first of each name first.

Section

A named run of the file that fields point into.

Spec

One format, as its spec describes it.

SpecDocs

What a spec says of its files, for the Documentation view: its own words and the notes its fields carry. None of it changes how a file is read.

SpecError

A problem with a spec: where it is (file, line, column) and what was expected.

Variant

One layout a record can take, picked by the type field.

Enums

Amount

A size or a count: written in the spec, or read from a field.

ChecksumAlgo

The checksums a spec can name.

ChipKind

What a chip’s value is, for its color and whether plain text quotes it.

Chosen

How a spec was chosen for a file.

Codec

The codec of each block: one for all, or one a header field names.

Compression

How a block’s body is compressed.

Delta

A value stored as the change from the previous record’s.

Encoding

How text is stored.

Endian

Byte order.

Expected

A header value a file must hold to match: match.where.

Framing

How one record is told from the next.

Layout

How the records sit in what is opened.

LookupFormat

How a symbol list’s entries are told apart.

Meaning

What a field means beyond its stored value.

Route

What route decided about one local path.

TimeUnitSpec

The unit a time field counts in.

Type

What a field’s bytes hold.

Constants

FORMAT_ONLY

What a spec with no match says in place of its conditions.

MAX_FLATTEN

The most columns one field may be flattened into.

MAX_SIZE

The largest size a spec or a file may give one field, header or record, and the most values one field may hold. A size read from a file past this is refused rather than believed.

MAX_SPEC_BYTES

The most a spec file may hold, local or remote: far more than any spec needs, and a bound on what --format FILE reads before it knows what it read.

MAX_SPEC_SAID

MAX_SPEC_BYTES, as the user is told it.

PATH_VAR

The variable the search path is extended with, after the config directory.

Traits

SpecRecords

The rows a spec reads, however they are framed: what the table, a window of it, formats check and the fuzz target read through.

Functions

all_fields

Every field of the records: the common ones, then each variant’s.

chips_plain

Chips as one line of plain text, for the command line and the Info panel: magic MKTD · version 1 · glob *.bin *.dat.

chosen_words

Why spec read a file, for the Notes tab: matched by magic MKTD · version 1 when its glob or its magic chose it, chosen by --format FILE otherwise.

command

datui formats, or datui formats check SPEC [FILE]: what to print, and the exit code (non-zero when the check finds an error).

head_of

The first reach bytes of path, through its decompressor when it has one.

is_spec_name

What a spec name may be: namespaced, as acme.l2feed is, so it cannot be taken for a built-in format.

path_parts

The parts {name} and {name:%Y%m%d} of a [files] path, in order.

read

Read path with the spec choice holds, naming it named in what it says.

route

Whether, and with which spec, path is read. In order: --format FILE, then --format NAME, then a glob, then magic. A file whose name or bytes say it is a format datui reads already keeps opening that way.

search_path

The directories and files searched for specs, in order: the config directory’s formats, then $DATUI_FORMATS_PATH, then [formats] path from the config.

search_path_for

The search path config asks for: the config directory’s formats, then $DATUI_FORMATS_PATH, then its [formats] path.

symbols

The entries of a symbol list.