Module plan 

The op graph a model builder emits and a backend lowers, see spec/07-engine.md.

A graph names its activations as Vals whose size is a row count times a width, and the row count is one of the three a batch has: tokens, sequences or markers. Nothing in a graph knows the batch size. A backend lowers the graph once per Bucket, which fixes the row counts, and layout gives every value an offset in one arena, reusing space once a value’s last reader has run.

Structs

Graph

A model’s forward pass as ops over values.

Layout

Values packed into one arena, in f32 elements.

Shape

The shape of a value: rows rows of width f32.

Val

An activation in a graph, an index into Graph::vals.

Enums

Epilogue

What a GEMM does to each output element after the bias.

Op

One step of a graph. Every op runs on the rows the batch has, not the bucket’s padded count.

Rows

What a value has one row per.

Constants

ALIGN

Offsets are multiples of this, 64 bytes, a cache line.

Functions

layout

Lays the values of graph out in one arena for the row counts rows, by a greedy interval colouring over their live ranges. A value lives from the first op that touches it to the last, and the outputs live to the end. Two values whose ranges overlap never share space, so an op’s inputs and outputs never alias unless the op is in place on one value. Values are placed largest first at the lowest offset that fits.

Type Aliases

W

A weight, the index of a tensor in the checkpoint the graph was built from.