Struct Stages 

pub struct Stages<T: CubeType> { /* private fields */ }

The depth slots of one buffered walk and the operands they are filled from.

Implementations

impl Stages<OperandPair>

pub fn consume_async<S: AsyncStage>(&mut self, walk: &Walk, stage: &S)

A compute plane’s side of walk, whose stages the filling planes fill by TMA: each stage issued through stage, and its slot freed only once what it issued completes. That is a stage later, so the reads of the previous stage run on beside the ones just issued, where the ring holds more than one slot; a ring of one waits for them at once.

pub fn __expand_consume_async<S: AsyncStage>( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, walk: &<Walk as CubeType>::ExpandType, stage: &<S as CubeType>::ExpandType, )

A compute plane’s side of walk, whose stages the filling planes fill by TMA: each stage issued through stage, and its slot freed only once what it issued completes. That is a stage later, so the reads of the previous stage run on beside the ones just issued, where the ring holds more than one slot; a ring of one waits for them at once.

impl<T: CubeType> Stages<T>

pub fn role(&self) -> Role

What this plane does with the stages’ slots; filler planes sit at the end of the cube.

pub fn slot_mut(&mut self, index: usize) -> &mut Slot<T>

Slot index, known at comptime.

pub fn __expand_role( scope: &Scope, this: &<Self as CubeType>::ExpandType, ) -> <Role as CubeType>::ExpandType

What this plane does with the stages’ slots; filler planes sit at the end of the cube.

pub fn __expand_slot_mut<'infer, 'scope>( scope: &'scope Scope, this: &'infer mut <Self as CubeType>::ExpandType, index: usize, ) -> &'infer mut <Slot<T> as CubeType>::ExpandType

Slot index, known at comptime.

impl<P: Payload<P> + Clone + CubeType<ExpandType: Clone>> Stages<P>

pub fn free(&self)

Free the shared memory every slot holds, for what the kernel declares after it to take: once the walk is over, nothing reads a stage again. Every unit of the cube reaches it, and a cube barrier comes before any unit touches what is declared in its place, as Tile::smem_accumulation and Tile::smem_cyclic_accumulation open with.

pub fn fill(&mut self, slot: usize, region: &Region)

Fill slot slot from region’s window: one step of a Fill plane’s walk.

pub fn __expand_free(scope: &Scope, this: &<Self as CubeType>::ExpandType)

Free the shared memory every slot holds, for what the kernel declares after it to take: once the walk is over, nothing reads a stage again. Every unit of the cube reaches it, and a cube barrier comes before any unit touches what is declared in its place, as Tile::smem_accumulation and Tile::smem_cyclic_accumulation open with.

pub fn __expand_fill( scope: &Scope, this: &mut <Self as CubeType>::ExpandType, slot: usize, region: &<Region as CubeType>::ExpandType, )

Fill slot slot from region’s window: one step of a Fill plane’s walk.

impl Stages<OperandPair>

pub fn smem( walk: &Walk, lhs: &Tile, rhs: &Tile, storage: StageStorage, depth: usize, decoded: Option<ElemType>, ) -> Stages<OperandPair>

The depth slots staging both operands of a contraction into shared memory laid out as storage.

A stage whose fill decodes its operand, from scales, a codebook or fields as wide as their word, holds the values at decoded, or at f32 where None. An operand staged as it lies keeps its own element.

pub fn __expand_smem( scope: &Scope, walk: &<Walk as CubeType>::ExpandType, lhs: &<Tile as CubeType>::ExpandType, rhs: &<Tile as CubeType>::ExpandType, storage: StageStorage, depth: usize, decoded: Option<ElemType>, ) -> <Stages<OperandPair> as CubeType>::ExpandType

The depth slots staging both operands of a contraction into shared memory laid out as storage.

A stage whose fill decodes its operand, from scales, a codebook or fields as wide as their word, holds the values at decoded, or at f32 where None. An operand staged as it lies keeps its own element.

impl Stages<Tile>

pub fn smem_single( walk: &Walk, input: &Tile, storage: StageStorage, depth: usize, decoded: Option<ElemType>, ) -> Stages<Tile>

smem for the sole operand input.

pub fn smem_single_at( walk: &Walk, input: &Tile, storage: StageStorage, width: Option<usize>, depth: usize, decoded: Option<ElemType>, ) -> Stages<Tile>

smem_single with the stage served in width-wide lines.

pub fn __expand_smem_single( scope: &Scope, walk: &<Walk as CubeType>::ExpandType, input: &<Tile as CubeType>::ExpandType, storage: StageStorage, depth: usize, decoded: Option<ElemType>, ) -> <Stages<Tile> as CubeType>::ExpandType

smem for the sole operand input.

pub fn __expand_smem_single_at( scope: &Scope, walk: &<Walk as CubeType>::ExpandType, input: &<Tile as CubeType>::ExpandType, storage: StageStorage, width: Option<usize>, depth: usize, decoded: Option<ElemType>, ) -> <Stages<Tile> as CubeType>::ExpandType

smem_single with the stage served in width-wide lines.

impl Stages<OperandPair>

pub fn consume(&mut self, _slot: usize, _compute: impl FnOnce(&Tile, &Tile))

Consume slot slot: one step of a Compute plane’s walk.

impl Stages<Tile>

pub fn consume(&mut self, _slot: usize, _compute: impl FnOnce(&Tile))

consume for the sole operand.

impl<T: CubeType> Stages<T>

pub fn pipelined<F>(&mut self, _walk: Walk, _compute: F)

where F: FnMut(&mut Slot<T>, &Region),

Walk walk‘s regions, filling slots from the stages’ sources and compute consuming each through consume.

impl Stages<OperandPair>

pub fn walk<F>(&mut self, _walk: Walk, _prefetch: Prefetch, _compute: F)

where F: FnMut(&mut Slot<OperandPair>, &Region),

Walk walk’s stages with the schedule prefetch selects.

impl Stages<OperandPair>

pub fn prefetched<F>(&mut self, _walk: Walk, _compute: F)

where F: FnMut(&mut Slot<OperandPair>, &Region),

pipelined with the next region’s loads held in registers across the current contraction.

Requires depth 1 or 2, every unit copying straight from a plain operand, and loads within Prefetch::fits; otherwise refused at expansion. Fixed operands, and operands whose stages keep their scales beside them, fall back to pipelined.

Trait Implementations

impl<T: CubeType> CubeType for Stages<T>

type ExpandType = StagesExpand<T>