Struct TileExpand 

pub struct TileExpand { /* private fields */ }

Implementations

impl TileExpand

pub fn __expand_mm_method( &mut self, scope: &Scope, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, )

c = a · b, under the semiring the accumulator was opened with, or a memory window stated accumulating. A memory window starts the sum fresh rather than reading the cell.

pub fn __expand_dequantized_method( &self, scope: &Scope, operand: &<Tile as CubeType>::ExpandType, element: ElemType, ) -> <Tile as CubeType>::ExpandType

operand dequantized at element for this accumulator to contract: the plane-level dequantize, called where the kernel wants it instead of left to the leaf, which loads it as the multiply would. A cmma grid loads it into its fragments through a landing; one cmma fragment into the fragment its leaf builds; a walk of steps into the landing the steps read, where the tensor cores read through one. A plain operand is returned as it is: it has nothing to decode. So is any operand of a manual-mma sum, whose contraction decodes it as it loads it, at its other factor’s element (Tile::mma): loaded here, ahead of that factor, its fragments could not hold k in the order the other factor’s do.

pub fn __expand_mma_method( &mut self, scope: &Scope, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, )

c += a · b, added to what the accumulator or memory window holds. A packed factor is decoded into f32, unless the kernel dequantized it at another element first, a manual-mma instruction decodes it at its other factor’s element, or the instruction reads its codes as stored.

pub fn __expand_reset_method(&mut self, scope: &Scope)

This plane-resident accumulator back at its semiring’s identity, as it was opened.

pub fn __expand_mm( scope: &Scope, this: &mut Self, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, )

c = a · b, under the semiring the accumulator was opened with, or a memory window stated accumulating. A memory window starts the sum fresh rather than reading the cell.

pub fn __expand_dequantized( scope: &Scope, this: &Self, operand: &<Tile as CubeType>::ExpandType, element: ElemType, ) -> <Tile as CubeType>::ExpandType

operand dequantized at element for this accumulator to contract: the plane-level dequantize, called where the kernel wants it instead of left to the leaf, which loads it as the multiply would. A cmma grid loads it into its fragments through a landing; one cmma fragment into the fragment its leaf builds; a walk of steps into the landing the steps read, where the tensor cores read through one. A plain operand is returned as it is: it has nothing to decode. So is any operand of a manual-mma sum, whose contraction decodes it as it loads it, at its other factor’s element (Tile::mma): loaded here, ahead of that factor, its fragments could not hold k in the order the other factor’s do.

pub fn __expand_mma( scope: &Scope, this: &mut Self, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, )

c += a · b, added to what the accumulator or memory window holds. A packed factor is decoded into f32, unless the kernel dequantized it at another element first, a manual-mma instruction decodes it at its other factor’s element, or the instruction reads its codes as stored.

pub fn __expand_reset(scope: &Scope, this: &mut Self)

This plane-resident accumulator back at its semiring’s identity, as it was opened.

impl TileExpand

pub fn __expand_commit_method( &self, scope: &Scope, ) -> <Pending<()> as CubeType>::ExpandType

Commit the warpgroup MMAs this accumulator issued since its last commit, and return their completion: once it resolves, they are done reading what they read, the stages a walk holds for them among it. Only a warpgroup accumulator contracts asynchronously.

pub fn __expand_commit( scope: &Scope, this: &Self, ) -> <Pending<()> as CubeType>::ExpandType

Commit the warpgroup MMAs this accumulator issued since its last commit, and return their completion: once it resolves, they are done reading what they read, the stages a walk holds for them among it. Only a warpgroup accumulator contracts asynchronously.

impl TileExpand

pub fn __expand_accumulating_method( &self, scope: &Scope, block: RegisterBlock, semiring: Semiring, ) -> <Tile as CubeType>::ExpandType

This memory window as a contraction’s output: each unit’s share tiled into block, summed under semiring. A plane-resident accumulator states both when it is opened.

pub fn __expand_accumulating( scope: &Scope, this: &Self, block: RegisterBlock, semiring: Semiring, ) -> <Tile as CubeType>::ExpandType

This memory window as a contraction’s output: each unit’s share tiled into block, summed under semiring. A plane-resident accumulator states both when it is opened.

impl TileExpand

pub fn __expand_product_method( &mut self, scope: &Scope, a: &<Tile as CubeType>::ExpandType, b: &<Tile as CubeType>::ExpandType, )

dst = a ⊗ b, elementwise over the destination’s box.

pub fn __expand_product( scope: &Scope, this: &mut Self, a: &<Tile as CubeType>::ExpandType, b: &<Tile as CubeType>::ExpandType, )

dst = a ⊗ b, elementwise over the destination’s box.

impl TileExpand

pub fn __expand_reduce_axis_method( &mut self, scope: &Scope, input: &<Tile as CubeType>::ExpandType, monoid: Monoid, )

c = reduce(input): reduce input into self across the contracted axes.

pub fn __expand_dot_method<E: Float>( &mut self, scope: &Scope, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, )

self[c] = Σ_r lhs[c, r] · rhs[c, r], r the axes lhs and rhs span beyond self’s: each cell the dot product of its run of both, as attention’s backward pass sums its output against the output’s gradient along the value dim. self is a window of memory a unit or a plane holds (Tile::set_cells), every unit of the holder summing every cell; lhs and rhs span one space, bound at one width.

pub fn __expand_reduce_axis_accumulate_method( &mut self, scope: &Scope, input: &<Tile as CubeType>::ExpandType, monoid: Monoid, )

c = reduce(c, input): reduce input into what self holds, which must be seeded.

pub fn __expand_reduce_axis( scope: &Scope, this: &mut Self, input: &<Tile as CubeType>::ExpandType, monoid: Monoid, )

c = reduce(input): reduce input into self across the contracted axes.

pub fn __expand_dot<E: Float>( scope: &Scope, this: &mut Self, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, )

self[c] = Σ_r lhs[c, r] · rhs[c, r], r the axes lhs and rhs span beyond self’s: each cell the dot product of its run of both, as attention’s backward pass sums its output against the output’s gradient along the value dim. self is a window of memory a unit or a plane holds (Tile::set_cells), every unit of the holder summing every cell; lhs and rhs span one space, bound at one width.

pub fn __expand_reduce_axis_accumulate( scope: &Scope, this: &mut Self, input: &<Tile as CubeType>::ExpandType, monoid: Monoid, )

c = reduce(c, input): reduce input into what self holds, which must be seeded.

impl TileExpand

pub fn __expand_accumulator_method<EA: Numeric>( &self, scope: &Scope, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, instruction: Instruction, semiring: Semiring, ) -> <Tile as CubeType>::ExpandType

What one plane sums lhs · rhs into under semiring, in the form instruction names, at the element EA, opened at the semiring’s identity.

pub fn __expand_block_reducer_method<EA: Numeric>( &self, scope: &Scope, input: &<Tile as CubeType>::ExpandType, config: RegisterBlock, monoid: Monoid, ) -> <Tile as CubeType>::ExpandType

A register block over this tile’s grid for a reduction over input under monoid, at the element EA, opened at its identity.

pub fn __expand_with_scratch_method( self, scope: &Scope, scratch: Scratch, ) -> <Tile as CubeType>::ExpandType

This accumulator opened with a per-plane shared-memory scratch for bouncing drains, or as it is for Scratch::None. drained_into opens the smallest one itself where it needs one; stating it chooses how much of the grid it holds, or serves a drain region by region.

pub fn __expand_drained_into_method( &self, scope: &Scope, dest: &<Tile as CubeType>::ExpandType, )

Drain this accumulator into dest, cast to its element. self and dest must be indexed by the same region. A cmma grid opened with no scratch opens one where dest needs its fragments bounced.

pub fn __expand_accumulator<EA: Numeric>( scope: &Scope, this: &Self, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, instruction: Instruction, semiring: Semiring, ) -> <Tile as CubeType>::ExpandType

What one plane sums lhs · rhs into under semiring, in the form instruction names, at the element EA, opened at the semiring’s identity.

pub fn __expand_block_reducer<EA: Numeric>( scope: &Scope, this: &Self, input: &<Tile as CubeType>::ExpandType, config: RegisterBlock, monoid: Monoid, ) -> <Tile as CubeType>::ExpandType

A register block over this tile’s grid for a reduction over input under monoid, at the element EA, opened at its identity.

pub fn __expand_with_scratch( scope: &Scope, this: Self, scratch: Scratch, ) -> <Tile as CubeType>::ExpandType

This accumulator opened with a per-plane shared-memory scratch for bouncing drains, or as it is for Scratch::None. drained_into opens the smallest one itself where it needs one; stating it chooses how much of the grid it holds, or serves a drain region by region.

pub fn __expand_drained_into( scope: &Scope, this: &Self, dest: &<Tile as CubeType>::ExpandType, )

Drain this accumulator into dest, cast to its element. self and dest must be indexed by the same region. A cmma grid opened with no scratch opens one where dest needs its fragments bounced.

impl TileExpand

pub fn __expand_cast_method<E: Numeric>( &self, scope: &Scope, ) -> <Tile as CubeType>::ExpandType

This plane-resident accumulator with each cell cast to E where its units hold it: what an attention contracts its probabilities with the values through, the next contraction reading the cast accumulator in its registers, without a round trip through shared memory.

Only a manual-mma accumulator is cast in place; any other is cast as it drains.

pub fn __expand_cast<E: Numeric>( scope: &Scope, this: &Self, ) -> <Tile as CubeType>::ExpandType

This plane-resident accumulator with each cell cast to E where its units hold it: what an attention contracts its probabilities with the values through, the next contraction reading the cast accumulator in its registers, without a round trip through shared memory.

Only a manual-mma accumulator is cast in place; any other is cast as it drains.

impl TileExpand

pub fn __expand_planes_output_method<A: Numeric>( &self, scope: &Scope, cube: &<Region as CubeType>::ExpandType, ) -> <PlanesOutput<A> as CubeType>::ExpandType

This tile, cube’s window of an output, as its planes reach it, a sum shared by planes added in A.

pub fn __expand_planes_output<A: Numeric>( scope: &Scope, this: &Self, cube: &<Region as CubeType>::ExpandType, ) -> <PlanesOutput<A> as CubeType>::ExpandType

This tile, cube’s window of an output, as its planes reach it, a sum shared by planes added in A.

impl TileExpand

pub fn __expand_smem_accumulation_method<A: Numeric>( &self, scope: &Scope, ) -> <SmemAccumulation as CubeType>::ExpandType

A zeroed shared-memory accumulator over this tile’s box, summing in A, with the cube synchronized on both sides of the zeroing.

pub fn __expand_smem_accumulation<A: Numeric>( scope: &Scope, this: &Self, ) -> <SmemAccumulation as CubeType>::ExpandType

A zeroed shared-memory accumulator over this tile’s box, summing in A, with the cube synchronized on both sides of the zeroing.

impl TileExpand

pub fn __expand_smem_cyclic_accumulation_method<A: Numeric>( &self, scope: &Scope, writers: usize, ) -> <SmemCyclicAccumulation<A> as CubeType>::ExpandType

A shared-memory accumulator over this tile’s box, summing in A, that writers writers reach exclusively, one chunk each a round. Nothing is zeroed: every cell’s first writer replaces it. It is served in lines as wide as this tile’s, so a partial drains into it and its sum lands in this tile line for line.

pub fn __expand_smem_cyclic_accumulation<A: Numeric>( scope: &Scope, this: &Self, writers: usize, ) -> <SmemCyclicAccumulation<A> as CubeType>::ExpandType

A shared-memory accumulator over this tile’s box, summing in A, that writers writers reach exclusively, one chunk each a round. Nothing is zeroed: every cell’s first writer replaces it. It is served in lines as wide as this tile’s, so a partial drains into it and its sum lands in this tile line for line.

impl TileExpand

pub fn __expand_smem_slots_accumulation_method<A: Numeric>( &self, scope: &Scope, writers: usize, ) -> <SmemSlotsAccumulation<A> as CubeType>::ExpandType

A shared-memory accumulator over this tile’s box, summing in A, with a slot for each of writers writers. Nothing is zeroed: every writer replaces its whole slot. It is served in lines as wide as this tile’s, so a partial drains into it and its sum lands in this tile line for line.

pub fn __expand_smem_slots_accumulation<A: Numeric>( scope: &Scope, this: &Self, writers: usize, ) -> <SmemSlotsAccumulation<A> as CubeType>::ExpandType

A shared-memory accumulator over this tile’s box, summing in A, with a slot for each of writers writers. Nothing is zeroed: every writer replaces its whole slot. It is served in lines as wide as this tile’s, so a partial drains into it and its sum lands in this tile line for line.

impl TileExpand

pub fn __expand_space_method(&self, scope: &Scope) -> Space

This tile’s own box at this depth.

pub fn __expand_with_levels_method( &self, scope: &Scope, depth: usize, levels: Vec<Level>, ) -> <Tile as CubeType>::ExpandType

These cells placed at depth of levels: a tile whose grid the caller states.

pub fn __expand_delivery_method(&self, scope: &Scope) -> Delivery

Who moves this operand’s bytes into a stage. Panics on a plane fragment.

pub fn __expand_vector_size_method(&self, scope: &Scope) -> usize

Physical vector width of the backing store; 1 for a fragment or tma source. Refuses a memory whose loads span several axes: its reader asks for the vector tile instead.

pub fn __expand_gathered_method(&self, scope: &Scope) -> bool

Whether this tile’s mapping has overlapping windows, so only an N-D view describes it.

pub fn __expand_is_shared_method(&self, scope: &Scope) -> bool

Whether this tile is backed by shared memory, the only space a sync_cube() orders.

pub fn __expand_within_method( &self, scope: &Scope, axis: Axis, from: <usize as CubeType>::ExpandType, until: <usize as CubeType>::ExpandType, ) -> <Tile as CubeType>::ExpandType

This operand’s window at from on axis, reading zero past until (both in elements).

pub fn __expand_read_page_entry_method( &self, scope: &Scope, region: &<Region as CubeType>::ExpandType, ) -> <PageEntry as CubeType>::ExpandType

Cuts region and returns the page table entry that window was cut with, loaded from the table unless an entry handed to this tile already named it. It is PageEntry::unlisted for a tile not read through a page table.

pub fn __expand_with_page_entry_method( &self, scope: &Scope, entry: &<PageEntry as CubeType>::ExpandType, ) -> <Tile as CubeType>::ExpandType

This tile, its cuts taking their page from entry where they name its sequence and column, and from the table otherwise. entry must come from a tile reading the same table, keys handing theirs to values included: it is neither checked nor clamped to the pool, so an entry from another table reads the page that table lists. A tile not read through a page table is unchanged.

pub fn __expand_at_method( &self, scope: &Scope, region: &<Region as CubeType>::ExpandType, ) -> <Tile as CubeType>::ExpandType

Window this tile down to region, no copy.

pub fn __expand_walk_method( &self, scope: &Scope, ) -> <Walk as CubeType>::ExpandType

The regions of the level below this tile over its own box.

pub fn __expand_over_method( &self, scope: &Scope, level: &Level, ) -> <Walk as CubeType>::ExpandType

The regions of level over this tile’s own box.

pub fn __expand_zero_method(&mut self, scope: &Scope)

Zero this whole tile.

pub fn __expand_scale_by_method<E: Numeric>( &mut self, scope: &Scope, factor: <E as CubeType>::ExpandType, )

Multiply every partial this accumulator holds by factor, cast to its element.

pub fn __expand_init_method<E: Numeric>( &mut self, scope: &Scope, val: <E as CubeType>::ExpandType, )

Initialize this tile with val, cast to its element.

pub fn __expand_stage_method( &self, scope: &Scope, level: Level, storage: StageStorage, ) -> <Tile as CubeType>::ExpandType

A fresh tile shaped to stage one region of level of this operand, laid out as storage.

pub fn __expand_stage_for_method( &self, scope: &Scope, walk: &<Walk as CubeType>::ExpandType, storage: StageStorage, decoded: Option<ElemType>, ) -> <Tile as CubeType>::ExpandType

A fresh stage of this operand for one region of walk, laid out as storage: shaped, owned and placed as the walk’s stages are, the cube’s or each plane’s own copy. What a kernel stages once before a walk that leaves the operand unchanged, as an attention’s query beside the keys it walks.

A stage whose fill decodes its operand holds the values at decoded, or at f32 where None, as Stages::smem does. An operand staged as it lies keeps its own element.

pub fn __expand_smem_box_method( &self, scope: &Scope, ) -> <Tile as CubeType>::ExpandType

A shared-memory tile over this tile’s box, one element a line, its rows dense and in order, at this tile’s place in its nest: what a bulk store moves out whole (copy_from into a tile a tensor map backs), and what an accumulator drains into first, as it would into this tile. It starts on Delivery::TMA_STAGE_ALIGNMENT, which a launch sizing its shared memory counts.

pub fn __expand_copy_from_method( &mut self, scope: &Scope, src: &<Tile as CubeType>::ExpandType, )

Move src into self, the kind pairing picking the instruction.

Into a tile a tensor map backs, from a shared-memory box (smem_box), it is a bulk store that every unit of the cube reaches: each fences the writes it made to the box, the cube synchronizes, one unit stores it and waits for the engine to have read it, and the cube synchronizes again before anything writes the box.

A source whose scales a stage keeps beside it copies its values as they are stored, into a stage that keeps them too; the stage’s fill brings the scales. Any other scaled source is written with its scales applied, which only dequantize_from does.

pub fn __expand_dequantize_from_method( &mut self, scope: &Scope, src: &<Tile as CubeType>::ExpandType, )

Writes src’s numbers into this tile, every scale it carries applied and its table looked up: the one copy that applies a tile’s scales.

From memory, each load is decoded once, in the unit that reads it, its scales multiplied together in f32 and the value rounded once. From an accumulator, it is the sum’s drain, which multiplies the sum by the scale it carries; the sum is spent by its write.

pub fn __expand_quantize_from_method( &mut self, scope: &Scope, src: &<Tile as CubeType>::ExpandType, )

Writes src’s numbers into this tile packed, under the block scales this tile carries, which it computes: the mirror of dequantize_from. The tile’s packing says the format its values are written in, and its innermost scale tile the dtype its scales are stored as; a scale coarser than the block’s (a per-tensor one) is read, as the caller already wrote it.

Calibrated as burn’s quantize_dynamic is: a block’s scale is its largest magnitude over the format’s largest, rounded up to the scale’s dtype, relative to the coarser scale where there is one; each value is divided by its scale and rounded to the nearest code.

pub fn __expand_add_cast_from_method( &mut self, scope: &Scope, src: &<Tile as CubeType>::ExpandType, )

Add a resident block src into this one, casting up to T, without touching memory.

pub fn __expand_copy_cast_from_method( &mut self, scope: &Scope, src: &<Tile as CubeType>::ExpandType, )

copy_from with a cast: stores a wider resident fragment down to T, or copies a memory window of the same box, the units sharing it a line at a time.

pub fn __expand_space(scope: &Scope, this: &Self) -> Space

This tile’s own box at this depth.

pub fn __expand_with_levels( scope: &Scope, this: &Self, depth: usize, levels: Vec<Level>, ) -> <Tile as CubeType>::ExpandType

These cells placed at depth of levels: a tile whose grid the caller states.

pub fn __expand_delivery(scope: &Scope, this: &Self) -> Delivery

Who moves this operand’s bytes into a stage. Panics on a plane fragment.

pub fn __expand_vector_size(scope: &Scope, this: &Self) -> usize

Physical vector width of the backing store; 1 for a fragment or tma source. Refuses a memory whose loads span several axes: its reader asks for the vector tile instead.

pub fn __expand_gathered(scope: &Scope, this: &Self) -> bool

Whether this tile’s mapping has overlapping windows, so only an N-D view describes it.

pub fn __expand_is_shared(scope: &Scope, this: &Self) -> bool

Whether this tile is backed by shared memory, the only space a sync_cube() orders.

pub fn __expand_within( scope: &Scope, this: &Self, axis: Axis, from: <usize as CubeType>::ExpandType, until: <usize as CubeType>::ExpandType, ) -> <Tile as CubeType>::ExpandType

This operand’s window at from on axis, reading zero past until (both in elements).

pub fn __expand_read_page_entry( scope: &Scope, this: &Self, region: &<Region as CubeType>::ExpandType, ) -> <PageEntry as CubeType>::ExpandType

Cuts region and returns the page table entry that window was cut with, loaded from the table unless an entry handed to this tile already named it. It is PageEntry::unlisted for a tile not read through a page table.

pub fn __expand_with_page_entry( scope: &Scope, this: &Self, entry: &<PageEntry as CubeType>::ExpandType, ) -> <Tile as CubeType>::ExpandType

This tile, its cuts taking their page from entry where they name its sequence and column, and from the table otherwise. entry must come from a tile reading the same table, keys handing theirs to values included: it is neither checked nor clamped to the pool, so an entry from another table reads the page that table lists. A tile not read through a page table is unchanged.

pub fn __expand_at( scope: &Scope, this: &Self, region: &<Region as CubeType>::ExpandType, ) -> <Tile as CubeType>::ExpandType

Window this tile down to region, no copy.

pub fn __expand_walk( scope: &Scope, this: &Self, ) -> <Walk as CubeType>::ExpandType

The regions of the level below this tile over its own box.

pub fn __expand_over( scope: &Scope, this: &Self, level: &Level, ) -> <Walk as CubeType>::ExpandType

The regions of level over this tile’s own box.

pub fn __expand_zero(scope: &Scope, this: &mut Self)

Zero this whole tile.

pub fn __expand_scale_by<E: Numeric>( scope: &Scope, this: &mut Self, factor: <E as CubeType>::ExpandType, )

Multiply every partial this accumulator holds by factor, cast to its element.

pub fn __expand_init<E: Numeric>( scope: &Scope, this: &mut Self, val: <E as CubeType>::ExpandType, )

Initialize this tile with val, cast to its element.

pub fn __expand_stage( scope: &Scope, this: &Self, level: Level, storage: StageStorage, ) -> <Tile as CubeType>::ExpandType

A fresh tile shaped to stage one region of level of this operand, laid out as storage.

pub fn __expand_stage_for( scope: &Scope, this: &Self, walk: &<Walk as CubeType>::ExpandType, storage: StageStorage, decoded: Option<ElemType>, ) -> <Tile as CubeType>::ExpandType

A fresh stage of this operand for one region of walk, laid out as storage: shaped, owned and placed as the walk’s stages are, the cube’s or each plane’s own copy. What a kernel stages once before a walk that leaves the operand unchanged, as an attention’s query beside the keys it walks.

A stage whose fill decodes its operand holds the values at decoded, or at f32 where None, as Stages::smem does. An operand staged as it lies keeps its own element.

pub fn __expand_scratch<T: Numeric>( scope: &Scope, walk: &<Walk as CubeType>::ExpandType, axes: Vec<Axis>, storage: StageStorage, ) -> <Tile as CubeType>::ExpandType

A fresh shared-memory tile over axes of one region of walk, laid out as storage and served one value a line: placed where the walk’s regions sit, and owned as the walk’s stages are, one for the cube or a copy for each plane (Stages).

What a kernel holds between two steps of its walk that no operand is: an attention’s scores between its two contractions. Placed in the partitioning, it opens an accumulator and is windowed by the walk’s regions like an operand.

pub fn __expand_shared<T: Numeric>( scope: &Scope, space: Space, storage: StageStorage, ) -> <Tile as CubeType>::ExpandType

A fresh shared-memory tile over space, laid out as storage, serving one value a line.

pub fn __expand_smem<T: Numeric>( scope: &Scope, space: Space, vector_size: usize, storage: StageStorage, units: usize, ) -> <Tile as CubeType>::ExpandType

A fresh shared-memory tile over space, served vector_size wide and filled by units units (0: the whole cube).

pub fn __expand_smem_box( scope: &Scope, this: &Self, ) -> <Tile as CubeType>::ExpandType

A shared-memory tile over this tile’s box, one element a line, its rows dense and in order, at this tile’s place in its nest: what a bulk store moves out whole (copy_from into a tile a tensor map backs), and what an accumulator drains into first, as it would into this tile. It starts on Delivery::TMA_STAGE_ALIGNMENT, which a launch sizing its shared memory counts.

pub fn __expand_copy_from( scope: &Scope, this: &mut Self, src: &<Tile as CubeType>::ExpandType, )

Move src into self, the kind pairing picking the instruction.

Into a tile a tensor map backs, from a shared-memory box (smem_box), it is a bulk store that every unit of the cube reaches: each fences the writes it made to the box, the cube synchronizes, one unit stores it and waits for the engine to have read it, and the cube synchronizes again before anything writes the box.

A source whose scales a stage keeps beside it copies its values as they are stored, into a stage that keeps them too; the stage’s fill brings the scales. Any other scaled source is written with its scales applied, which only dequantize_from does.

pub fn __expand_dequantize_from( scope: &Scope, this: &mut Self, src: &<Tile as CubeType>::ExpandType, )

Writes src’s numbers into this tile, every scale it carries applied and its table looked up: the one copy that applies a tile’s scales.

From memory, each load is decoded once, in the unit that reads it, its scales multiplied together in f32 and the value rounded once. From an accumulator, it is the sum’s drain, which multiplies the sum by the scale it carries; the sum is spent by its write.

pub fn __expand_quantize_from( scope: &Scope, this: &mut Self, src: &<Tile as CubeType>::ExpandType, )

Writes src’s numbers into this tile packed, under the block scales this tile carries, which it computes: the mirror of dequantize_from. The tile’s packing says the format its values are written in, and its innermost scale tile the dtype its scales are stored as; a scale coarser than the block’s (a per-tensor one) is read, as the caller already wrote it.

Calibrated as burn’s quantize_dynamic is: a block’s scale is its largest magnitude over the format’s largest, rounded up to the scale’s dtype, relative to the coarser scale where there is one; each value is divided by its scale and rounded to the nearest code.

pub fn __expand_add_cast_from( scope: &Scope, this: &mut Self, src: &<Tile as CubeType>::ExpandType, )

Add a resident block src into this one, casting up to T, without touching memory.

pub fn __expand_copy_cast_from( scope: &Scope, this: &mut Self, src: &<Tile as CubeType>::ExpandType, )

copy_from with a cast: stores a wider resident fragment down to T, or copies a memory window of the same box, the units sharing it a line at a time.

impl TileExpand

pub fn space(&self) -> Space

This tile’s own box, readable inside comptime!.

impl TileExpand

pub fn __expand_mul_method( &self, scope: &Scope, scale: &TileExpand, ) -> TileExpand

pub fn __expand_factor_method( &self, scope: &Scope, space: Space, instruction: Instruction, ) -> (TileExpand, FactorExpand)

pub fn __expand_scaled_method( &self, _scope: &Scope, factor: &FactorExpand, ) -> TileExpand

pub fn __expand_lookup_method( &self, _scope: &Scope, table: &TileExpand, ) -> TileExpand

impl TileExpand

pub fn __expand_cells_method<E: Float>( &self, scope: &Scope, count: usize, ) -> <Array<E> as CubeType>::ExpandType

This tile’s cells in order, one value a slice of something else, as the rows of a softmax: a window of memory a unit or a plane holds, count cells, every unit of the holder reading them all.

pub fn __expand_set_cells_method<E: Float>( &mut self, scope: &Scope, values: &<Array<E> as CubeType>::ExpandType, count: usize, )

Write values over this tile’s count cells (cells): a unit writes its own window, and a plane’s is written by its first unit.

pub fn __expand_cells<E: Float>( scope: &Scope, this: &Self, count: usize, ) -> <Array<E> as CubeType>::ExpandType

This tile’s cells in order, one value a slice of something else, as the rows of a softmax: a window of memory a unit or a plane holds, count cells, every unit of the holder reading them all.

pub fn __expand_set_cells<E: Float>( scope: &Scope, this: &mut Self, values: &<Array<E> as CubeType>::ExpandType, count: usize, )

Write values over this tile’s count cells (cells): a unit writes its own window, and a plane’s is written by its first unit.

impl TileExpand

pub fn __expand_entry_in_range_method( &self, scope: &Scope, ) -> NativeExpand<bool>

impl TileExpand

pub fn __expand_view_method<'infer, 'scope, T: Numeric, W: Size>( &'infer self, scope: &'scope Scope, ) -> <View<'infer, Vector<T, W>, CoordsDyn> as CubeType>::ExpandType

A read View over Vector<T, W> lines through the base layout and Window. W is the line width (self.store.vector_size).

pub fn __expand_view_mut_method<'infer, 'scope, T: Numeric, W: Size>( &'infer mut self, scope: &'scope Scope, ) -> <ViewMut<'infer, Vector<T, W>, CoordsDyn> as CubeType>::ExpandType

pub fn __expand_view<'infer, 'scope, T: Numeric, W: Size>( scope: &'scope Scope, this: &'infer Self, ) -> <View<'infer, Vector<T, W>, CoordsDyn> as CubeType>::ExpandType

A read View over Vector<T, W> lines through the base layout and Window. W is the line width (self.store.vector_size).

pub fn __expand_view_mut<'infer, 'scope, T: Numeric, W: Size>( scope: &'scope Scope, this: &'infer mut Self, ) -> <ViewMut<'infer, Vector<T, W>, CoordsDyn> as CubeType>::ExpandType

impl TileExpand

pub fn __expand_decoded_method<O: Float, W: Size>( &self, scope: &Scope, ) -> <Decoded<O, W> as CubeType>::ExpandType

This tile’s window read into registers a line at a time, W values a line, each served at O decoded as dequantize_from writes it: its fields unpacked, looked up in the table they index, under the scales they carry. What a unit reading the lines it needs straight out of memory takes, with no stage to decode into.

Plain values under no scale read as flat reads them. Packed values are read a word a load, so they are bound one word wide; a line takes the words it spans, or the fields it needs of one, and their scale, which must not vary along the line.

Panics

As the kernel expands: plain values under scales, which only a copy decodes; packed ones bound wider than a word, or whose scales vary along a line.

pub fn __expand_decoded<O: Float, W: Size>( scope: &Scope, this: &Self, ) -> <Decoded<O, W> as CubeType>::ExpandType

This tile’s window read into registers a line at a time, W values a line, each served at O decoded as dequantize_from writes it: its fields unpacked, looked up in the table they index, under the scales they carry. What a unit reading the lines it needs straight out of memory takes, with no stage to decode into.

Plain values under no scale read as flat reads them. Packed values are read a word a load, so they are bound one word wide; a line takes the words it spans, or the fields it needs of one, and their scale, which must not vary along the line.

Panics

As the kernel expands: plain values under scales, which only a copy decodes; packed ones bound wider than a word, or whose scales vary along a line.

impl TileExpand

pub fn __expand_flat_method<'infer, 'scope, T: Numeric, W: Size>( &'infer self, scope: &'scope Scope, ) -> <Masked<'infer, Vector<T, W>, Coords1d> as CubeType>::ExpandType

A flat row-major view over Vector<T, W> lines of the tile’s window, masking the overhang. A packed store is refused.

pub fn __expand_flat_mut_method<'infer, 'scope, T: Numeric, W: Size>( &'infer mut self, scope: &'scope Scope, ) -> <MaskedMut<'infer, Vector<T, W>, Coords1d> as CubeType>::ExpandType

The mutable twin of flat.

pub fn __expand_flat<'infer, 'scope, T: Numeric, W: Size>( scope: &'scope Scope, this: &'infer Self, ) -> <Masked<'infer, Vector<T, W>, Coords1d> as CubeType>::ExpandType

A flat row-major view over Vector<T, W> lines of the tile’s window, masking the overhang. A packed store is refused.

pub fn __expand_flat_mut<'infer, 'scope, T: Numeric, W: Size>( scope: &'scope Scope, this: &'infer mut Self, ) -> <MaskedMut<'infer, Vector<T, W>, Coords1d> as CubeType>::ExpandType

The mutable twin of flat.

impl TileExpand

pub fn __expand_values_method( &self, scope: &Scope, ) -> <MemoryValues as CubeType>::ExpandType

This tile’s values, read one at a time (MemoryValues).

Panics

At expansion, on a tile that is not memory.

pub fn __expand_values( scope: &Scope, this: &Self, ) -> <MemoryValues as CubeType>::ExpandType

This tile’s values, read one at a time (MemoryValues).

Panics

At expansion, on a tile that is not memory.

impl TileExpand

pub fn __expand_along_method( &self, scope: &Scope, axis: Axis, ) -> <AxisSlices as CubeType>::ExpandType

This tile cut into slices along axis, where its holder keeps them: a unit’s register block, a plane’s window of shared memory, or a plane’s fragments. A window of memory any wider holder shares has no one to read a slice whole, and is refused.

pub fn __expand_along( scope: &Scope, this: &Self, axis: Axis, ) -> <AxisSlices as CubeType>::ExpandType

This tile cut into slices along axis, where its holder keeps them: a unit’s register block, a plane’s window of shared memory, or a plane’s fragments. A window of memory any wider holder shares has no one to read a slice whole, and is refused.

impl TileExpand

pub fn __expand_slices_per_unit_method( &self, scope: &Scope, axis: Axis, ) -> usize

Slices along axis one unit keeps a state for: the rows a manual-mma grid’s unit holds cells of, which a reduction reads in its registers; every slice of any other holder.

pub fn __expand_slices_per_unit(scope: &Scope, this: &Self, axis: Axis) -> usize

Slices along axis one unit keeps a state for: the rows a manual-mma grid’s unit holds cells of, which a reduction reads in its registers; every slice of any other holder.

Trait Implementations

impl AsMutExpand for TileExpand

fn __expand_ref_mut_method(&mut self, _: &Scope) -> &mut Self

fn __expand_as_mut_method(&mut self, scope: &Scope) -> &mut T

impl AsRefExpand for TileExpand

fn __expand_ref_method(&self, _: &Scope) -> &Self

fn __expand_as_ref_method(&self, scope: &Scope) -> &T

impl Clone for TileExpand

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl CubeDebug for TileExpand

fn set_debug_name(&self, scope: &Scope, name: &'static str)

Set the debug name of this type’s expansion. Should do nothing for types that don’t appear at runtime

impl DerefExpand for TileExpand

where for<'__cube_deref> <TileKind as CubeType>::ExpandType: DerefExpand<Target = <TileKind as CubeType>::ExpandType>, for<'__cube_deref> <Factor as CubeType>::ExpandType: DerefExpand<Target = <Factor as CubeType>::ExpandType>,

type Target = TileExpand

fn __expand_deref_method(&self, scope: &Scope) -> Self

impl ExpandTypeClone for TileExpand

fn clone_unchecked(&self) -> Self

Unchecked clone that only clones the conceptual runtime value. Should only be used in cases where each copy is used in a mutually exclusive branch (i.e. match, runtime enums). This is intentionally separated from Rust’s Clone semantics and should only be used for the conceptual expand values, never real data. Using two values in the same branch is undefined behaviour.

impl IntoExpand for TileExpand

type Expand = TileExpand

fn into_expand(self, _: &Scope) -> Self

impl IntoMut for TileExpand

fn into_mut(self, scope: &Scope) -> Self

Convert the variable into a potentially new mutable variable in scope, copying if needed.

impl Iterable for TileExpand

for plane in tile iterates the level below the tile over its own box.

type Item = RegionExpand

fn expand(self, scope: &Scope, body: &mut dyn FnMut(&Scope, RegionExpand))

Expand a runtime loop without unrolling

fn expand_unroll( self, scope: &Scope, body: &mut dyn FnMut(&Scope, RegionExpand), )

Expand an unrolled loop. The body should be invoced n times, where n is the number of iterations.

fn const_len(&self) -> Option<usize>

Return the comptime length of this iterable, if possible

impl Iterable for &TileExpand

for plane in &tile.

type Item = RegionExpand

fn expand(self, scope: &Scope, body: &mut dyn FnMut(&Scope, RegionExpand))

Expand a runtime loop without unrolling

fn expand_unroll( self, scope: &Scope, body: &mut dyn FnMut(&Scope, RegionExpand), )

Expand an unrolled loop. The body should be invoced n times, where n is the number of iterations.

fn const_len(&self) -> Option<usize>

Return the comptime length of this iterable, if possible