Struct Barrier 

pub struct Barrier;

A mechanism for awaiting on asynchronous data transfers Behaviour is defined by its BarrierLevel.

Implementations

impl Barrier

pub fn tma_load_1d<C: CubePrimitive>( &self, source: &TensorMap<C>, destination: &mut SliceMut<Line<C>>, x: i32, )

Copy a tile from a global memory source to a shared memory destination, with the provided offsets.

pub fn __expand_tma_load_1d<C: CubePrimitive>( scope: &mut Scope, expand: BarrierExpand, source: ExpandElementTyped<TensorMap<C>>, destination: SliceExpand<Line<C>, ReadWrite>, x: ExpandElementTyped<i32>, )

impl Barrier

pub fn tma_load_2d<C: CubePrimitive>( &self, source: &TensorMap<C>, destination: &mut SliceMut<Line<C>>, y: i32, x: i32, )

Copy a tile from a global memory source to a shared memory destination, with the provided offsets.

pub fn __expand_tma_load_2d<C: CubePrimitive>( scope: &mut Scope, expand: BarrierExpand, source: ExpandElementTyped<TensorMap<C>>, destination: SliceExpand<Line<C>, ReadWrite>, y: ExpandElementTyped<i32>, x: ExpandElementTyped<i32>, )

impl Barrier

pub fn tma_load_3d<C: CubePrimitive>( &self, source: &TensorMap<C>, destination: &mut SliceMut<Line<C>>, z: i32, y: i32, x: i32, )

Copy a tile from a global memory source to a shared memory destination, with the provided offsets.

pub fn __expand_tma_load_3d<C: CubePrimitive>( scope: &mut Scope, expand: BarrierExpand, source: ExpandElementTyped<TensorMap<C>>, destination: SliceExpand<Line<C>, ReadWrite>, z: ExpandElementTyped<i32>, y: ExpandElementTyped<i32>, x: ExpandElementTyped<i32>, )

impl Barrier

pub fn tma_load_4d<C: CubePrimitive>( &self, source: &TensorMap<C>, destination: &mut SliceMut<Line<C>>, w: i32, z: i32, y: i32, x: i32, )

Copy a tile from a global memory source to a shared memory destination, with the provided offsets.

pub fn __expand_tma_load_4d<C: CubePrimitive>( scope: &mut Scope, expand: BarrierExpand, source: ExpandElementTyped<TensorMap<C>>, destination: SliceExpand<Line<C>, ReadWrite>, w: ExpandElementTyped<i32>, z: ExpandElementTyped<i32>, y: ExpandElementTyped<i32>, x: ExpandElementTyped<i32>, )

impl Barrier

pub fn tma_load_5d<C: CubePrimitive>( &self, source: &TensorMap<C>, destination: &mut SliceMut<Line<C>>, v: i32, w: i32, z: i32, y: i32, x: i32, )

Copy a tile from a global memory source to a shared memory destination, with the provided offsets.

pub fn __expand_tma_load_5d<C: CubePrimitive>( scope: &mut Scope, expand: BarrierExpand, source: ExpandElementTyped<TensorMap<C>>, destination: SliceExpand<Line<C>, ReadWrite>, v: ExpandElementTyped<i32>, w: ExpandElementTyped<i32>, z: ExpandElementTyped<i32>, y: ExpandElementTyped<i32>, x: ExpandElementTyped<i32>, )

impl Barrier

pub fn tma_load_im2col_3d<C: CubePrimitive>( &self, source: &TensorMap<C>, destination: &mut SliceMut<Line<C>>, n: i32, w: i32, c: i32, w_offset: u16, )

Copy a tile from a global memory source to a shared memory destination, with the provided offsets.

pub fn __expand_tma_load_im2col_3d<C: CubePrimitive>( scope: &mut Scope, expand: BarrierExpand, source: ExpandElementTyped<TensorMap<C>>, destination: SliceExpand<Line<C>, ReadWrite>, n: ExpandElementTyped<i32>, w: ExpandElementTyped<i32>, c: ExpandElementTyped<i32>, w_offset: ExpandElementTyped<u16>, )

impl Barrier

pub fn tma_load_im2col_4d<C: CubePrimitive>( &self, source: &TensorMap<C>, destination: &mut SliceMut<Line<C>>, n: i32, h: i32, w: i32, c: i32, h_offset: u16, w_offset: u16, )

Copy a tile from a global memory source to a shared memory destination, with the provided offsets.

pub fn __expand_tma_load_im2col_4d<C: CubePrimitive>( scope: &mut Scope, expand: BarrierExpand, source: ExpandElementTyped<TensorMap<C>>, destination: SliceExpand<Line<C>, ReadWrite>, n: ExpandElementTyped<i32>, h: ExpandElementTyped<i32>, w: ExpandElementTyped<i32>, c: ExpandElementTyped<i32>, h_offset: ExpandElementTyped<u16>, w_offset: ExpandElementTyped<u16>, )

impl Barrier

pub fn tma_load_im2col_5d<C: CubePrimitive>( &self, source: &TensorMap<C>, destination: &mut SliceMut<Line<C>>, n: i32, d: i32, h: i32, w: i32, c: i32, d_offset: u16, h_offset: u16, w_offset: u16, )

Copy a tile from a global memory source to a shared memory destination, with the provided offsets.

pub fn __expand_tma_load_im2col_5d<C: CubePrimitive>( scope: &mut Scope, expand: BarrierExpand, source: ExpandElementTyped<TensorMap<C>>, destination: SliceExpand<Line<C>, ReadWrite>, n: ExpandElementTyped<i32>, d: ExpandElementTyped<i32>, h: ExpandElementTyped<i32>, w: ExpandElementTyped<i32>, c: ExpandElementTyped<i32>, d_offset: ExpandElementTyped<u16>, h_offset: ExpandElementTyped<u16>, w_offset: ExpandElementTyped<u16>, )

impl Barrier

pub fn local() -> Self

Create a local barrier object for the current unit. Automatically initialized with an arrival count of 1.

pub fn shared(arrival_count: u32, is_elected: bool) -> Shared<Barrier>

Create a shared memory barrier that can be accesses by all units in the cube. Initialized by the is_elected unit with an arrival count of arrival_count. This is the number of times arrive or one of its variants needs to be called before the barrier advances.

If all units in the cube arrive on the barrier, use CUBE_DIM as the arrival count. For other purposes, only a subset may need to arrive.

pub fn shared_uninit() -> Shared<Barrier>

Create a shared memory barrier that can be accesses by all units in the cube. Only declared, but not initialized.

pub fn init_manual(&self, arrival_count: u32)

Initializes a barrier with a given arrival_count. This is the number of times arrive or one of its variants needs to be called before the barrier advances.

If all units in the cube arrive on the barrier, use CUBE_DIM as the arrival count. For other purposes, only a subset may need to arrive.

Note

No synchronization or election is performed, this is raw initialization. For shared barriers ensure only one unit performs the initialization, and synchronize the cube afterwards. There may also be additional synchronization requirements for bulk copy operations, like sync_async_proxy_shared().

pub fn __expand_local(scope: &mut Scope) -> <Self as CubeType>::ExpandType

pub fn __expand_shared( scope: &mut Scope, arrival_count: <u32 as CubeType>::ExpandType, is_elected: <bool as CubeType>::ExpandType, ) -> <Shared<Barrier> as CubeType>::ExpandType

pub fn __expand_shared_uninit( scope: &mut Scope, ) -> <Shared<Barrier> as CubeType>::ExpandType

pub fn __expand_init_manual( scope: &mut Scope, this: &<Self as CubeType>::ExpandType, arrival_count: <u32 as CubeType>::ExpandType, ) -> <() as CubeType>::ExpandType

impl Barrier

pub fn memcpy_async<C: CubePrimitive>( &self, source: &Slice<Line<C>>, destination: &mut SliceMut<Line<C>>, )

Copy the source slice to destination

Safety

This will try to copy the whole source slice, so make sure source length <= destination length

pub fn memcpy_async_cooperative<C: CubePrimitive>( &self, source: &Slice<Line<C>>, destination: &mut SliceMut<Line<C>>, )

Copy the source slice to destination

Safety

This will try to copy the whole source slice, so make sure source length <= destination length

pub fn memcpy_async_tx<C: CubePrimitive>( &self, source: &Slice<Line<C>>, destination: &mut SliceMut<Line<C>>, )

Copy the source slice to destination. Uses transaction count like TMA, so use with expect_tx or arrive_and_expect_tx.

Safety

This will try to copy the whole source slice, so make sure source length <= destination length

pub fn __expand_memcpy_async<C: CubePrimitive>( scope: &mut Scope, this: &<Self as CubeType>::ExpandType, source: <Slice<Line<C>> as CubeType>::ExpandType, destination: <SliceMut<Line<C>> as CubeType>::ExpandType, ) -> <() as CubeType>::ExpandType

pub fn __expand_memcpy_async_cooperative<C: CubePrimitive>( scope: &mut Scope, this: &<Self as CubeType>::ExpandType, source: <Slice<Line<C>> as CubeType>::ExpandType, destination: <SliceMut<Line<C>> as CubeType>::ExpandType, ) -> <() as CubeType>::ExpandType

pub fn __expand_memcpy_async_tx<C: CubePrimitive>( scope: &mut Scope, this: &<Self as CubeType>::ExpandType, source: <Slice<Line<C>> as CubeType>::ExpandType, destination: <SliceMut<Line<C>> as CubeType>::ExpandType, ) -> <() as CubeType>::ExpandType

impl Barrier

pub fn arrive(&self) -> BarrierToken

Arrive at the barrier, decrementing arrival count

pub fn arrive_and_expect_tx( &self, arrival_count: u32, transaction_count: u32, ) -> BarrierToken

Arrive at the barrier, decrementing arrival count. Additionally increments expected count.

pub fn expect_tx(&self, expected_count: u32)

Increments the expected count of the barrier.

pub fn arrive_and_wait(&self)

Wait until all data is loaded

pub fn wait(&self, token: BarrierToken)

Wait at the barrier until all arrivals are done

pub fn wait_parity(&self, phase: u32)

Wait at the barrier until the phase is completed. Doesn’t require a token, but needs phase to be managed manually.

pub fn __expand_arrive( scope: &mut Scope, this: &<Self as CubeType>::ExpandType, ) -> <BarrierToken as CubeType>::ExpandType

pub fn __expand_arrive_and_expect_tx( scope: &mut Scope, this: &<Self as CubeType>::ExpandType, arrival_count: <u32 as CubeType>::ExpandType, transaction_count: <u32 as CubeType>::ExpandType, ) -> <BarrierToken as CubeType>::ExpandType

pub fn __expand_expect_tx( scope: &mut Scope, this: &<Self as CubeType>::ExpandType, expected_count: <u32 as CubeType>::ExpandType, ) -> <() as CubeType>::ExpandType

pub fn __expand_arrive_and_wait( scope: &mut Scope, this: &<Self as CubeType>::ExpandType, ) -> <() as CubeType>::ExpandType

pub fn __expand_wait( scope: &mut Scope, this: &<Self as CubeType>::ExpandType, token: <BarrierToken as CubeType>::ExpandType, ) -> <() as CubeType>::ExpandType

pub fn __expand_wait_parity( scope: &mut Scope, this: &<Self as CubeType>::ExpandType, phase: <u32 as CubeType>::ExpandType, ) -> <() as CubeType>::ExpandType

impl Barrier

pub fn commit_copy_async(&self)

Makes all previous copy_async operations visible on the barrier. Should be called once after all copies have been dispatched, before reading from the shared memory.

Does not count as an arrive in terms of the barrier arrival count. So arrive or arrive_and_wait should still be called afterwards.

pub fn __expand_commit_copy_async( scope: &mut Scope, this: &<Self as CubeType>::ExpandType, ) -> <() as CubeType>::ExpandType

Trait Implementations

impl Clone for Barrier

fn clone(&self) -> Barrier

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl CubePrimitive for Barrier

fn from_const_value(_value: ConstantScalarValue) -> Self

fn as_type(_scope: &Scope) -> StorageType

Return the element type to use on GPU.

fn as_type_native() -> Option<StorageType>

Native or static element type.

fn as_type_native_unchecked() -> StorageType

Native or static element type.

fn size() -> Option<usize>

Only native element types have a size.

fn size_bits() -> Option<usize>

Only native element types have a size.

fn size_bits_unchecked() -> usize

Only native element types have a size.

fn from_expand_elem(elem: ExpandElement) -> Self::ExpandType

fn into_lit_unchecked(self) -> Self

fn supported_uses<R: Runtime>(client: &ComputeClient<R>) -> EnumSet<TypeUsage>

fn type_size() -> u32

fn type_size_bits() -> u32

fn packing_factor() -> u32

fn __expand_type_size(scope: &Scope) -> u32

fn __expand_type_size_bits(scope: &Scope) -> u32

fn __expand_packing_factor(scope: &Scope) -> u32

impl CubeType for Barrier

type ExpandType = ExpandElementTyped<Barrier>

fn into_mut(scope: &mut Scope, expand: Self::ExpandType) -> Self::ExpandType

Wrapper around the init method, necessary to type inference.

impl ExpandElementIntoMut for Barrier

fn elem_into_mut(_scope: &mut Scope, elem: ExpandElement) -> ExpandElement

impl PartialEq for Barrier

fn eq(&self, other: &Barrier) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for Barrier

impl Eq for Barrier

impl StructuralPartialEq for Barrier