Struct Tile 

pub struct Tile { /* private fields */ }

One operand’s data: a runtime backing store and the comptime Space it projects.

Implementations

impl Tile

pub fn mm(&mut self, lhs: &Tile, rhs: &Tile)

c = a · b, under the semiring the accumulator was opened with, or a memory window stated accumulating. A memory window starts the sum fresh rather than reading the cell.

pub fn dequantized(&self, operand: &Tile, element: ElemType) -> Tile

operand dequantized at element for this accumulator to contract: the plane-level dequantize, called where the kernel wants it instead of left to the leaf, which loads it as the multiply would. A cmma grid loads it into its fragments through a landing; one cmma fragment into the fragment its leaf builds; a walk of steps into the landing the steps read, where the tensor cores read through one. A plain operand is returned as it is: it has nothing to decode. So is any operand of a manual-mma sum, whose contraction decodes it as it loads it, at its other factor’s element (Tile::mma): loaded here, ahead of that factor, its fragments could not hold k in the order the other factor’s do.

pub fn mma(&mut self, lhs: &Tile, rhs: &Tile)

c += a · b, added to what the accumulator or memory window holds. A packed factor is decoded into f32, unless the kernel dequantized it at another element first, a manual-mma instruction decodes it at its other factor’s element, or the instruction reads its codes as stored.

pub fn reset(&mut self)

This plane-resident accumulator back at its semiring’s identity, as it was opened.

pub fn __expand_mm( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, )

c = a · b, under the semiring the accumulator was opened with, or a memory window stated accumulating. A memory window starts the sum fresh rather than reading the cell.

pub fn __expand_dequantized( scope: &Scope, this: &<Self as CubeType>::ExpandType, operand: &<Tile as CubeType>::ExpandType, element: ElemType, ) -> <Tile as CubeType>::ExpandType

operand dequantized at element for this accumulator to contract: the plane-level dequantize, called where the kernel wants it instead of left to the leaf, which loads it as the multiply would. A cmma grid loads it into its fragments through a landing; one cmma fragment into the fragment its leaf builds; a walk of steps into the landing the steps read, where the tensor cores read through one. A plain operand is returned as it is: it has nothing to decode. So is any operand of a manual-mma sum, whose contraction decodes it as it loads it, at its other factor’s element (Tile::mma): loaded here, ahead of that factor, its fragments could not hold k in the order the other factor’s do.

pub fn __expand_mma( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, )

c += a · b, added to what the accumulator or memory window holds. A packed factor is decoded into f32, unless the kernel dequantized it at another element first, a manual-mma instruction decodes it at its other factor’s element, or the instruction reads its codes as stored.

pub fn __expand_reset(scope: &Scope, this: &mut <Self as CubeType>::ExpandType)

This plane-resident accumulator back at its semiring’s identity, as it was opened.

impl Tile

pub fn commit(&self) -> Pending<()>

Commit the warpgroup MMAs this accumulator issued since its last commit, and return their completion: once it resolves, they are done reading what they read, the stages a walk holds for them among it. Only a warpgroup accumulator contracts asynchronously.

pub fn __expand_commit( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> <Pending<()> as CubeType>::ExpandType

Commit the warpgroup MMAs this accumulator issued since its last commit, and return their completion: once it resolves, they are done reading what they read, the stages a walk holds for them among it. Only a warpgroup accumulator contracts asynchronously.

impl Tile

pub fn accumulating(&self, block: RegisterBlock, semiring: Semiring) -> Tile

This memory window as a contraction’s output: each unit’s share tiled into block, summed under semiring. A plane-resident accumulator states both when it is opened.

pub fn __expand_accumulating( scope: &Scope, this: &<Self as CubeType>::ExpandType, block: RegisterBlock, semiring: Semiring, ) -> <Tile as CubeType>::ExpandType

This memory window as a contraction’s output: each unit’s share tiled into block, summed under semiring. A plane-resident accumulator states both when it is opened.

impl Tile

pub fn product(&mut self, a: &Tile, b: &Tile)

dst = a ⊗ b, elementwise over the destination’s box.

pub fn __expand_product( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, a: &<Tile as CubeType>::ExpandType, b: &<Tile as CubeType>::ExpandType, )

dst = a ⊗ b, elementwise over the destination’s box.

impl Tile

pub fn reduce_axis(&mut self, input: &Tile, monoid: Monoid)

c = reduce(input): reduce input into self across the contracted axes.

pub fn dot<E: Float>(&mut self, lhs: &Tile, rhs: &Tile)

self[c] = Σ_r lhs[c, r] · rhs[c, r], r the axes lhs and rhs span beyond self’s: each cell the dot product of its run of both, as attention’s backward pass sums its output against the output’s gradient along the value dim. self is a window of memory a unit or a plane holds (Tile::set_cells), every unit of the holder summing every cell; lhs and rhs span one space, bound at one width.

pub fn reduce_axis_accumulate(&mut self, input: &Tile, monoid: Monoid)

c = reduce(c, input): reduce input into what self holds, which must be seeded.

pub fn __expand_reduce_axis( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, input: &<Tile as CubeType>::ExpandType, monoid: Monoid, )

c = reduce(input): reduce input into self across the contracted axes.

pub fn __expand_dot<E: Float>( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, )

self[c] = Σ_r lhs[c, r] · rhs[c, r], r the axes lhs and rhs span beyond self’s: each cell the dot product of its run of both, as attention’s backward pass sums its output against the output’s gradient along the value dim. self is a window of memory a unit or a plane holds (Tile::set_cells), every unit of the holder summing every cell; lhs and rhs span one space, bound at one width.

pub fn __expand_reduce_axis_accumulate( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, input: &<Tile as CubeType>::ExpandType, monoid: Monoid, )

c = reduce(c, input): reduce input into what self holds, which must be seeded.

impl Tile

pub fn accumulator<EA: Numeric>( &self, lhs: &Tile, rhs: &Tile, instruction: Instruction, semiring: Semiring, ) -> Tile

What one plane sums lhs · rhs into under semiring, in the form instruction names, at the element EA, opened at the semiring’s identity.

pub fn block_reducer<EA: Numeric>( &self, input: &Tile, config: RegisterBlock, monoid: Monoid, ) -> Tile

A register block over this tile’s grid for a reduction over input under monoid, at the element EA, opened at its identity.

pub fn with_scratch(self, scratch: Scratch) -> Tile

This accumulator opened with a per-plane shared-memory scratch for bouncing drains, or as it is for Scratch::None. drained_into opens the smallest one itself where it needs one; stating it chooses how much of the grid it holds, or serves a drain region by region.

pub fn drained_into(&self, dest: &Tile)

Drain this accumulator into dest, cast to its element. self and dest must be indexed by the same region. A cmma grid opened with no scratch opens one where dest needs its fragments bounced.

pub fn __expand_accumulator<EA: Numeric>( scope: &Scope, this: &<Self as CubeType>::ExpandType, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, instruction: Instruction, semiring: Semiring, ) -> <Tile as CubeType>::ExpandType

What one plane sums lhs · rhs into under semiring, in the form instruction names, at the element EA, opened at the semiring’s identity.

pub fn __expand_block_reducer<EA: Numeric>( scope: &Scope, this: &<Self as CubeType>::ExpandType, input: &<Tile as CubeType>::ExpandType, config: RegisterBlock, monoid: Monoid, ) -> <Tile as CubeType>::ExpandType

A register block over this tile’s grid for a reduction over input under monoid, at the element EA, opened at its identity.

pub fn __expand_with_scratch( scope: &Scope, this: <Self as CubeType>::ExpandType, scratch: Scratch, ) -> <Tile as CubeType>::ExpandType

This accumulator opened with a per-plane shared-memory scratch for bouncing drains, or as it is for Scratch::None. drained_into opens the smallest one itself where it needs one; stating it chooses how much of the grid it holds, or serves a drain region by region.

pub fn __expand_drained_into( scope: &Scope, this: &<Self as CubeType>::ExpandType, dest: &<Tile as CubeType>::ExpandType, )

Drain this accumulator into dest, cast to its element. self and dest must be indexed by the same region. A cmma grid opened with no scratch opens one where dest needs its fragments bounced.

impl Tile

pub fn cast<E: Numeric>(&self) -> Tile

This plane-resident accumulator with each cell cast to E where its units hold it: what an attention contracts its probabilities with the values through, the next contraction reading the cast accumulator in its registers, without a round trip through shared memory.

Only a manual-mma accumulator is cast in place; any other is cast as it drains.

pub fn __expand_cast<E: Numeric>( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> <Tile as CubeType>::ExpandType

This plane-resident accumulator with each cell cast to E where its units hold it: what an attention contracts its probabilities with the values through, the next contraction reading the cast accumulator in its registers, without a round trip through shared memory.

Only a manual-mma accumulator is cast in place; any other is cast as it drains.

impl Tile

pub fn stage_accumulator<EA: Numeric>( walk: &Walk, axes: Vec<Axis>, lhs: &Tile, instruction: Instruction, ) -> Tile

A plane-resident accumulator over one region of walk along axes, summing lhs times a factor in the matrix instruction instruction names, opened at zero: the accumulator Tile::scratch would open, without the memory, for a product contracted again before anything drains it, as an attention’s score.

pub fn __expand_stage_accumulator<EA: Numeric>( scope: &Scope, walk: &<Walk as CubeType>::ExpandType, axes: Vec<Axis>, lhs: &<Tile as CubeType>::ExpandType, instruction: Instruction, ) -> <Tile as CubeType>::ExpandType

A plane-resident accumulator over one region of walk along axes, summing lhs times a factor in the matrix instruction instruction names, opened at zero: the accumulator Tile::scratch would open, without the memory, for a product contracted again before anything drains it, as an attention’s score.

impl Tile

pub fn planes_output<A: Numeric>(&self, cube: &Region) -> PlanesOutput<A>

This tile, cube’s window of an output, as its planes reach it, a sum shared by planes added in A.

pub fn __expand_planes_output<A: Numeric>( scope: &Scope, this: &<Self as CubeType>::ExpandType, cube: &<Region as CubeType>::ExpandType, ) -> <PlanesOutput<A> as CubeType>::ExpandType

This tile, cube’s window of an output, as its planes reach it, a sum shared by planes added in A.

impl Tile

pub fn smem_accumulation<A: Numeric>(&self) -> SmemAccumulation

A zeroed shared-memory accumulator over this tile’s box, summing in A, with the cube synchronized on both sides of the zeroing.

pub fn __expand_smem_accumulation<A: Numeric>( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> <SmemAccumulation as CubeType>::ExpandType

A zeroed shared-memory accumulator over this tile’s box, summing in A, with the cube synchronized on both sides of the zeroing.

impl Tile

pub fn smem_cyclic_accumulation<A: Numeric>( &self, writers: usize, ) -> SmemCyclicAccumulation<A>

A shared-memory accumulator over this tile’s box, summing in A, that writers writers reach exclusively, one chunk each a round. Nothing is zeroed: every cell’s first writer replaces it. It is served in lines as wide as this tile’s, so a partial drains into it and its sum lands in this tile line for line.

pub fn __expand_smem_cyclic_accumulation<A: Numeric>( scope: &Scope, this: &<Self as CubeType>::ExpandType, writers: usize, ) -> <SmemCyclicAccumulation<A> as CubeType>::ExpandType

A shared-memory accumulator over this tile’s box, summing in A, that writers writers reach exclusively, one chunk each a round. Nothing is zeroed: every cell’s first writer replaces it. It is served in lines as wide as this tile’s, so a partial drains into it and its sum lands in this tile line for line.

impl Tile

pub fn smem_slots_accumulation<A: Numeric>( &self, writers: usize, ) -> SmemSlotsAccumulation<A>

A shared-memory accumulator over this tile’s box, summing in A, with a slot for each of writers writers. Nothing is zeroed: every writer replaces its whole slot. It is served in lines as wide as this tile’s, so a partial drains into it and its sum lands in this tile line for line.

pub fn __expand_smem_slots_accumulation<A: Numeric>( scope: &Scope, this: &<Self as CubeType>::ExpandType, writers: usize, ) -> <SmemSlotsAccumulation<A> as CubeType>::ExpandType

A shared-memory accumulator over this tile’s box, summing in A, with a slot for each of writers writers. Nothing is zeroed: every writer replaces its whole slot. It is served in lines as wide as this tile’s, so a partial drains into it and its sum lands in this tile line for line.

impl Tile

pub fn space(&self) -> Space

This tile’s own box at this depth.

pub fn with_levels(&self, depth: usize, levels: Vec<Level>) -> Tile

These cells placed at depth of levels: a tile whose grid the caller states.

pub fn delivery(&self) -> Delivery

Who moves this operand’s bytes into a stage. Panics on a plane fragment.

pub fn vector_size(&self) -> usize

Physical vector width of the backing store; 1 for a fragment or tma source. Refuses a memory whose loads span several axes: its reader asks for the vector tile instead.

pub fn gathered(&self) -> bool

Whether this tile’s mapping has overlapping windows, so only an N-D view describes it.

pub fn is_shared(&self) -> bool

Whether this tile is backed by shared memory, the only space a sync_cube() orders.

pub fn within(&self, axis: Axis, from: usize, until: usize) -> Tile

This operand’s window at from on axis, reading zero past until (both in elements).

pub fn read_page_entry(&self, region: &Region) -> PageEntry

Cuts region and returns the page table entry that window was cut with, loaded from the table unless an entry handed to this tile already named it. It is PageEntry::unlisted for a tile not read through a page table.

pub fn with_page_entry(&self, entry: &PageEntry) -> Tile

This tile, its cuts taking their page from entry where they name its sequence and column, and from the table otherwise. entry must come from a tile reading the same table, keys handing theirs to values included: it is neither checked nor clamped to the pool, so an entry from another table reads the page that table lists. A tile not read through a page table is unchanged.

pub fn at(&self, region: &Region) -> Tile

Window this tile down to region, no copy.

pub fn walk(&self) -> Walk

The regions of the level below this tile over its own box.

pub fn over(&self, level: &Level) -> Walk

The regions of level over this tile’s own box.

pub fn zero(&mut self)

Zero this whole tile.

pub fn scale_by<E: Numeric>(&mut self, factor: E)

Multiply every partial this accumulator holds by factor, cast to its element.

pub fn init<E: Numeric>(&mut self, val: E)

Initialize this tile with val, cast to its element.

pub fn stage(&self, level: Level, storage: StageStorage) -> Tile

A fresh tile shaped to stage one region of level of this operand, laid out as storage.

pub fn stage_for( &self, walk: &Walk, storage: StageStorage, decoded: Option<ElemType>, ) -> Tile

A fresh stage of this operand for one region of walk, laid out as storage: shaped, owned and placed as the walk’s stages are, the cube’s or each plane’s own copy. What a kernel stages once before a walk that leaves the operand unchanged, as an attention’s query beside the keys it walks.

A stage whose fill decodes its operand holds the values at decoded, or at f32 where None, as Stages::smem does. An operand staged as it lies keeps its own element.

pub fn scratch<T: Numeric>( walk: &Walk, axes: Vec<Axis>, storage: StageStorage, ) -> Tile

A fresh shared-memory tile over axes of one region of walk, laid out as storage and served one value a line: placed where the walk’s regions sit, and owned as the walk’s stages are, one for the cube or a copy for each plane (Stages).

What a kernel holds between two steps of its walk that no operand is: an attention’s scores between its two contractions. Placed in the partitioning, it opens an accumulator and is windowed by the walk’s regions like an operand.

pub fn shared<T: Numeric>(space: Space, storage: StageStorage) -> Tile

A fresh shared-memory tile over space, laid out as storage, serving one value a line.

pub fn smem<T: Numeric>( space: Space, vector_size: usize, storage: StageStorage, units: usize, ) -> Tile

A fresh shared-memory tile over space, served vector_size wide and filled by units units (0: the whole cube).

pub fn smem_box(&self) -> Tile

A shared-memory tile over this tile’s box, one element a line, its rows dense and in order, at this tile’s place in its nest: what a bulk store moves out whole (copy_from into a tile a tensor map backs), and what an accumulator drains into first, as it would into this tile. It starts on Delivery::TMA_STAGE_ALIGNMENT, which a launch sizing its shared memory counts.

pub fn copy_from(&mut self, src: &Tile)

Move src into self, the kind pairing picking the instruction.

Into a tile a tensor map backs, from a shared-memory box (smem_box), it is a bulk store that every unit of the cube reaches: each fences the writes it made to the box, the cube synchronizes, one unit stores it and waits for the engine to have read it, and the cube synchronizes again before anything writes the box.

A source whose scales a stage keeps beside it copies its values as they are stored, into a stage that keeps them too; the stage’s fill brings the scales. Any other scaled source is written with its scales applied, which only dequantize_from does.

pub fn dequantize_from(&mut self, src: &Tile)

Writes src’s numbers into this tile, every scale it carries applied and its table looked up: the one copy that applies a tile’s scales.

From memory, each load is decoded once, in the unit that reads it, its scales multiplied together in f32 and the value rounded once. From an accumulator, it is the sum’s drain, which multiplies the sum by the scale it carries; the sum is spent by its write.

pub fn quantize_from(&mut self, src: &Tile)

Writes src’s numbers into this tile packed, under the block scales this tile carries, which it computes: the mirror of dequantize_from. The tile’s packing says the format its values are written in, and its innermost scale tile the dtype its scales are stored as; a scale coarser than the block’s (a per-tensor one) is read, as the caller already wrote it.

Calibrated as burn’s quantize_dynamic is: a block’s scale is its largest magnitude over the format’s largest, rounded up to the scale’s dtype, relative to the coarser scale where there is one; each value is divided by its scale and rounded to the nearest code.

pub fn add_cast_from(&mut self, src: &Tile)

Add a resident block src into this one, casting up to T, without touching memory.

pub fn copy_cast_from(&mut self, src: &Tile)

copy_from with a cast: stores a wider resident fragment down to T, or copies a memory window of the same box, the units sharing it a line at a time.

pub fn __expand_space( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> Space

This tile’s own box at this depth.

pub fn __expand_with_levels( scope: &Scope, this: &<Self as CubeType>::ExpandType, depth: usize, levels: Vec<Level>, ) -> <Tile as CubeType>::ExpandType

These cells placed at depth of levels: a tile whose grid the caller states.

pub fn __expand_delivery( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> Delivery

Who moves this operand’s bytes into a stage. Panics on a plane fragment.

pub fn __expand_vector_size( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> usize

Physical vector width of the backing store; 1 for a fragment or tma source. Refuses a memory whose loads span several axes: its reader asks for the vector tile instead.

pub fn __expand_gathered( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> bool

Whether this tile’s mapping has overlapping windows, so only an N-D view describes it.

pub fn __expand_is_shared( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> bool

Whether this tile is backed by shared memory, the only space a sync_cube() orders.

pub fn __expand_within( scope: &Scope, this: &<Self as CubeType>::ExpandType, axis: Axis, from: <usize as CubeType>::ExpandType, until: <usize as CubeType>::ExpandType, ) -> <Tile as CubeType>::ExpandType

This operand’s window at from on axis, reading zero past until (both in elements).

pub fn __expand_read_page_entry( scope: &Scope, this: &<Self as CubeType>::ExpandType, region: &<Region as CubeType>::ExpandType, ) -> <PageEntry as CubeType>::ExpandType

Cuts region and returns the page table entry that window was cut with, loaded from the table unless an entry handed to this tile already named it. It is PageEntry::unlisted for a tile not read through a page table.

pub fn __expand_with_page_entry( scope: &Scope, this: &<Self as CubeType>::ExpandType, entry: &<PageEntry as CubeType>::ExpandType, ) -> <Tile as CubeType>::ExpandType

This tile, its cuts taking their page from entry where they name its sequence and column, and from the table otherwise. entry must come from a tile reading the same table, keys handing theirs to values included: it is neither checked nor clamped to the pool, so an entry from another table reads the page that table lists. A tile not read through a page table is unchanged.

pub fn __expand_at( scope: &Scope, this: &<Self as CubeType>::ExpandType, region: &<Region as CubeType>::ExpandType, ) -> <Tile as CubeType>::ExpandType

Window this tile down to region, no copy.

pub fn __expand_walk( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> <Walk as CubeType>::ExpandType

The regions of the level below this tile over its own box.

pub fn __expand_over( scope: &Scope, this: &<Self as CubeType>::ExpandType, level: &Level, ) -> <Walk as CubeType>::ExpandType

The regions of level over this tile’s own box.

pub fn __expand_zero(scope: &Scope, this: &mut <Self as CubeType>::ExpandType)

Zero this whole tile.

pub fn __expand_scale_by<E: Numeric>( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, factor: <E as CubeType>::ExpandType, )

Multiply every partial this accumulator holds by factor, cast to its element.

pub fn __expand_init<E: Numeric>( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, val: <E as CubeType>::ExpandType, )

Initialize this tile with val, cast to its element.

pub fn __expand_stage( scope: &Scope, this: &<Self as CubeType>::ExpandType, level: Level, storage: StageStorage, ) -> <Tile as CubeType>::ExpandType

A fresh tile shaped to stage one region of level of this operand, laid out as storage.

pub fn __expand_stage_for( scope: &Scope, this: &<Self as CubeType>::ExpandType, walk: &<Walk as CubeType>::ExpandType, storage: StageStorage, decoded: Option<ElemType>, ) -> <Tile as CubeType>::ExpandType

A fresh stage of this operand for one region of walk, laid out as storage: shaped, owned and placed as the walk’s stages are, the cube’s or each plane’s own copy. What a kernel stages once before a walk that leaves the operand unchanged, as an attention’s query beside the keys it walks.

A stage whose fill decodes its operand holds the values at decoded, or at f32 where None, as Stages::smem does. An operand staged as it lies keeps its own element.

pub fn __expand_scratch<T: Numeric>( scope: &Scope, walk: &<Walk as CubeType>::ExpandType, axes: Vec<Axis>, storage: StageStorage, ) -> <Tile as CubeType>::ExpandType

A fresh shared-memory tile over axes of one region of walk, laid out as storage and served one value a line: placed where the walk’s regions sit, and owned as the walk’s stages are, one for the cube or a copy for each plane (Stages).

What a kernel holds between two steps of its walk that no operand is: an attention’s scores between its two contractions. Placed in the partitioning, it opens an accumulator and is windowed by the walk’s regions like an operand.

pub fn __expand_shared<T: Numeric>( scope: &Scope, space: Space, storage: StageStorage, ) -> <Tile as CubeType>::ExpandType

A fresh shared-memory tile over space, laid out as storage, serving one value a line.

pub fn __expand_smem<T: Numeric>( scope: &Scope, space: Space, vector_size: usize, storage: StageStorage, units: usize, ) -> <Tile as CubeType>::ExpandType

A fresh shared-memory tile over space, served vector_size wide and filled by units units (0: the whole cube).

pub fn __expand_smem_box( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> <Tile as CubeType>::ExpandType

A shared-memory tile over this tile’s box, one element a line, its rows dense and in order, at this tile’s place in its nest: what a bulk store moves out whole (copy_from into a tile a tensor map backs), and what an accumulator drains into first, as it would into this tile. It starts on Delivery::TMA_STAGE_ALIGNMENT, which a launch sizing its shared memory counts.

pub fn __expand_copy_from( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, src: &<Tile as CubeType>::ExpandType, )

Move src into self, the kind pairing picking the instruction.

Into a tile a tensor map backs, from a shared-memory box (smem_box), it is a bulk store that every unit of the cube reaches: each fences the writes it made to the box, the cube synchronizes, one unit stores it and waits for the engine to have read it, and the cube synchronizes again before anything writes the box.

A source whose scales a stage keeps beside it copies its values as they are stored, into a stage that keeps them too; the stage’s fill brings the scales. Any other scaled source is written with its scales applied, which only dequantize_from does.

pub fn __expand_dequantize_from( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, src: &<Tile as CubeType>::ExpandType, )

Writes src’s numbers into this tile, every scale it carries applied and its table looked up: the one copy that applies a tile’s scales.

From memory, each load is decoded once, in the unit that reads it, its scales multiplied together in f32 and the value rounded once. From an accumulator, it is the sum’s drain, which multiplies the sum by the scale it carries; the sum is spent by its write.

pub fn __expand_quantize_from( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, src: &<Tile as CubeType>::ExpandType, )

Writes src’s numbers into this tile packed, under the block scales this tile carries, which it computes: the mirror of dequantize_from. The tile’s packing says the format its values are written in, and its innermost scale tile the dtype its scales are stored as; a scale coarser than the block’s (a per-tensor one) is read, as the caller already wrote it.

Calibrated as burn’s quantize_dynamic is: a block’s scale is its largest magnitude over the format’s largest, rounded up to the scale’s dtype, relative to the coarser scale where there is one; each value is divided by its scale and rounded to the nearest code.

pub fn __expand_add_cast_from( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, src: &<Tile as CubeType>::ExpandType, )

Add a resident block src into this one, casting up to T, without touching memory.

pub fn __expand_copy_cast_from( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, src: &<Tile as CubeType>::ExpandType, )

copy_from with a cast: stores a wider resident fragment down to T, or copies a memory window of the same box, the units sharing it a line at a time.

impl Tile

pub fn mul(&self, _scale: &Tile) -> Tile

These values under scale, multiplied in where they are read. Every axis scale varies along must be one of the values’; one it spans without varying along is read at its origin.

pub fn factor(&self, _space: Space, _instruction: Instruction) -> (Tile, Factor)

This tile without its outer scales that vary only along axes of space, and those scales, for a sum instruction contracts. With space the output’s, they are the scales the sum takes out of every term and applies once (scaled), and the tile left carries the ones each term needs.

Words of block scales always give up the scales above them: the instruction reads the words undecoded, and holds no other scale. A decoded tile gives them up where the sum is scaled in place (Instruction::scales_its_sum_in_place), since a load converting codes in lane runs takes one level of scales; under any other sum they ride the load, which multiplies by them as it decodes, rather than send the sum through shared memory. A tile indexing a table gives up none: only its copy decodes it.

pub fn scaled(&self, _factor: &Factor) -> Tile

This tile under factor’s scales, outside its own.

impl Tile

pub fn cells<E: Float>(&self, count: usize) -> Array<E>

This tile’s cells in order, one value a slice of something else, as the rows of a softmax: a window of memory a unit or a plane holds, count cells, every unit of the holder reading them all.

pub fn set_cells<E: Float>(&mut self, values: &Array<E>, count: usize)

Write values over this tile’s count cells (cells): a unit writes its own window, and a plane’s is written by its first unit.

pub fn __expand_cells<E: Float>( scope: &Scope, this: &<Self as CubeType>::ExpandType, count: usize, ) -> <Array<E> as CubeType>::ExpandType

This tile’s cells in order, one value a slice of something else, as the rows of a softmax: a window of memory a unit or a plane holds, count cells, every unit of the holder reading them all.

pub fn __expand_set_cells<E: Float>( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, values: &<Array<E> as CubeType>::ExpandType, count: usize, )

Write values over this tile’s count cells (cells): a unit writes its own window, and a plane’s is written by its first unit.

impl Tile

pub fn entry_in_range(&self) -> bool

Whether the entry this tile’s table placed it with names a coordinate of the axis it places, or a run of at least one row: false for an entry past a stack of slabs, as a padding row’s route or a padding tile’s group names. Such a tile holds nothing, so a walk within it (Region::walk_within) takes no step. true for a tile with no table, or none read yet.

impl Tile

pub fn view<T: Numeric, W: Size>(&self) -> View<'_, Vector<T, W>, CoordsDyn>

A read View over Vector<T, W> lines through the base layout and Window. W is the line width (self.store.vector_size).

pub fn view_mut<T: Numeric, W: Size>( &mut self, ) -> ViewMut<'_, Vector<T, W>, CoordsDyn>

pub fn __expand_view<'infer, 'scope, T: Numeric, W: Size>( scope: &'scope Scope, this: &'infer <Self as CubeType>::ExpandType, ) -> <View<'infer, Vector<T, W>, CoordsDyn> as CubeType>::ExpandType

A read View over Vector<T, W> lines through the base layout and Window. W is the line width (self.store.vector_size).

pub fn __expand_view_mut<'infer, 'scope, T: Numeric, W: Size>( scope: &'scope Scope, this: &'infer mut <Self as CubeType>::ExpandType, ) -> <ViewMut<'infer, Vector<T, W>, CoordsDyn> as CubeType>::ExpandType

impl Tile

pub fn decoded<O: Float, W: Size>(&self) -> Decoded<O, W>

This tile’s window read into registers a line at a time, W values a line, each served at O decoded as dequantize_from writes it: its fields unpacked, looked up in the table they index, under the scales they carry. What a unit reading the lines it needs straight out of memory takes, with no stage to decode into.

Plain values under no scale read as flat reads them. Packed values are read a word a load, so they are bound one word wide; a line takes the words it spans, or the fields it needs of one, and their scale, which must not vary along the line.

Panics

As the kernel expands: plain values under scales, which only a copy decodes; packed ones bound wider than a word, or whose scales vary along a line.

pub fn __expand_decoded<O: Float, W: Size>( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> <Decoded<O, W> as CubeType>::ExpandType

This tile’s window read into registers a line at a time, W values a line, each served at O decoded as dequantize_from writes it: its fields unpacked, looked up in the table they index, under the scales they carry. What a unit reading the lines it needs straight out of memory takes, with no stage to decode into.

Plain values under no scale read as flat reads them. Packed values are read a word a load, so they are bound one word wide; a line takes the words it spans, or the fields it needs of one, and their scale, which must not vary along the line.

Panics

As the kernel expands: plain values under scales, which only a copy decodes; packed ones bound wider than a word, or whose scales vary along a line.

impl Tile

pub fn flat<T: Numeric, W: Size>(&self) -> Masked<'_, Vector<T, W>, Coords1d>

A flat row-major view over Vector<T, W> lines of the tile’s window, masking the overhang. A packed store is refused.

pub fn flat_mut<T: Numeric, W: Size>( &mut self, ) -> MaskedMut<'_, Vector<T, W>, Coords1d>

The mutable twin of flat.

pub fn __expand_flat<'infer, 'scope, T: Numeric, W: Size>( scope: &'scope Scope, this: &'infer <Self as CubeType>::ExpandType, ) -> <Masked<'infer, Vector<T, W>, Coords1d> as CubeType>::ExpandType

A flat row-major view over Vector<T, W> lines of the tile’s window, masking the overhang. A packed store is refused.

pub fn __expand_flat_mut<'infer, 'scope, T: Numeric, W: Size>( scope: &'scope Scope, this: &'infer mut <Self as CubeType>::ExpandType, ) -> <MaskedMut<'infer, Vector<T, W>, Coords1d> as CubeType>::ExpandType

The mutable twin of flat.

impl Tile

pub fn values(&self) -> MemoryValues

This tile’s values, read one at a time (MemoryValues).

Panics

At expansion, on a tile that is not memory.

pub fn __expand_values( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> <MemoryValues as CubeType>::ExpandType

This tile’s values, read one at a time (MemoryValues).

Panics

At expansion, on a tile that is not memory.

impl Tile

pub fn along(&self, axis: Axis) -> AxisSlices

This tile cut into slices along axis, where its holder keeps them: a unit’s register block, a plane’s window of shared memory, or a plane’s fragments. A window of memory any wider holder shares has no one to read a slice whole, and is refused.

pub fn __expand_along( scope: &Scope, this: &<Self as CubeType>::ExpandType, axis: Axis, ) -> <AxisSlices as CubeType>::ExpandType

This tile cut into slices along axis, where its holder keeps them: a unit’s register block, a plane’s window of shared memory, or a plane’s fragments. A window of memory any wider holder shares has no one to read a slice whole, and is refused.

impl Tile

pub fn slices_per_unit(&self, axis: Axis) -> usize

Slices along axis one unit keeps a state for: the rows a manual-mma grid’s unit holds cells of, which a reduction reads in its registers; every slice of any other holder.

pub fn __expand_slices_per_unit( scope: &Scope, this: &<Self as CubeType>::ExpandType, axis: Axis, ) -> usize

Slices along axis one unit keeps a state for: the rows a manual-mma grid’s unit holds cells of, which a reduction reads in its registers; every slice of any other holder.

Trait Implementations

impl Clone for Tile

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl CubeType for Tile

type ExpandType = TileExpand

impl IntoIterator for Tile

Host-side stub of for plane in tile; never runs.

type Item = Region

The type of the elements being iterated over.

type IntoIter = IntoIter<Region>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl IntoIterator for &Tile

type Item = Region

The type of the elements being iterated over.

type IntoIter = IntoIter<Region>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.