Crate presk_data 

The .psk.d data container and the training data pipeline.

A .psk.d file is a sequence of samples of one Presk type:

magic     "PSKD"                       4 bytes
version   u16 = 1                      little-endian
type_len  u32, then the type, UTF-8    the sample type in Presk syntax
count     u64                          number of samples
data      the samples, back to back: the leaves of the type in order,
          idx[N] as the smallest of u8/u16/u32/u64 holding N−1,
          ℝ as f64 little-endian

Version 2 (plan 0.17.0 step 7, D180) adds, after the count, one byte per leaf in storage order, the leaf’s encoding:

leaves    u32                          number of leaves of the type
encodings u8 per leaf                  0 f64 (and every idx), 1 f32, 2 bf16

and stores each real leaf in its encoding, little-endian (bf16 as the upper sixteen bits of an f32). Every number of f32 and of bf16 is a number of f64, so a leaf is read exactly whatever its encoding. A writer writes version 1 unless a leaf is narrower, so that a file of f64 leaves is the same bytes as before.

Real values are f64 and are injected exactly into ℝ when read (the hyperparameter rule of the language). The type is parsed with the Presk parser; a closed type (every dimension a literal) gives the leaf layout directly. A params M type needs the program to know its layout: the reader then returns the header only, and the caller supplies the Layout with Reader::with_layout.

Errors carry a code: E9001 (type not closed or not a data type), E9002 (not a .psk.d file), E9003 (truncated), E9004 (a value out of its bound or unparsable), E9005 (type mismatch).

Structs

DataError

Header

Pipeline

stream … |> shuffle(seed) |> batch(B) |> take(n) over sample indices.

Reader

A reader with random access to samples (fixed sample size).

Enums

Enc

How a real leaf’s elements are stored (version 2). Each is exactly a number of f64.

Layout

The leaf structure of a sample type, in the order the leaves are stored.

Leaf

One tensor leaf of a sample type.

Value

One sample: the same tree as its Layout, with flat row-major data per leaf.

Constants

MAGIC

VERSION

The version a file of f64 leaves is written in.

VERSION_ENCODED

The version of a file with a leaf narrower than f64.

Functions

char_vocab

The character vocabulary of a text: every distinct character, sorted, one token each. Tokens are Strings so that the same format can later hold sub-word tokens; index i denotes vocab[i].

decode_text

Decode a token stream back to text (the inverse of encode).

encode

Encode a text against a vocabulary. Only single-character vocabularies are supported: sub-word tokens need longest-match encoding, which comes with presk data bpe.

idx_width

The width of an idx[N] element: the smallest unsigned integer holding N−1.

info

presk data info: the header and the first n samples.

lm_windows

Language-modelling windows of a token stream: sample k is (tokens[k·stride .. +seq], tokens[k·stride + 1 .. +seq]), of type (idx[vocab][seq: T], idx[vocab][seq: T]).

shuffle_indices

A permutation of 0..n from seed and the epoch (1-based): the same on every machine, different for every epoch.

write

Writes a .psk.d file: sample_ty is the header text, layout its leaf structure (needed for params M, whose layout the text does not give).

write_encoded

[write], each real leaf in its encoding (encs in storage order, empty for every leaf in f64). The file is version 2 when a real leaf is narrower than f64, and version 1 otherwise, byte for byte what [write] writes. A value that is not a number of its leaf’s encoding is refused (E9004): the file holds the numbers it is given, exactly.

Type Aliases

Result