Struct LazyHostStorage 

pub struct LazyHostStorage { /* private fields */ }

Host storage for a tensor whose bytes are still on a device.

A device tensor crosses the model boundary wrapped in one of these. The result is a host tensor as far as facts, datum type and shape go, but nothing is copied back until something reads the bytes: as_bytes, as_slice, try_as_plain_ram and friends materialize it once and keep the result, and a caller that only hands it to the next run() never pays a readback at all, because DeviceSync takes the device tensor back out of it.

Only owned device tensors belong here. A view borrows a buffer whose contents are only good for the turn, so it is copied out at the boundary rather than held.

Implementations

impl LazyHostStorage

pub fn new(device: DeviceTensor) -> TractResult<Self>

Wrap a device tensor whose bytes are to be read later.

Waits for the work producing those bytes before wrapping: the wait belongs to the thread that queued the work, and a reader is free to be another one – a Metal stream is thread-local, so a wait deferred to the reader would be a wait on an empty queue and the bytes would be read before the device wrote them.

pub fn device(&self) -> Option<&DeviceTensor>

The device tensor still backing this storage, if it has not been detached by a host-side mutation.

pub fn is_materialized(&self) -> bool

True once the bytes have been brought back to host.

pub fn datum_type(&self) -> DatumType

pub fn shape(&self) -> TVec<usize>

pub fn into_tensor(self) -> Tensor

Wrap into a Tensor carrying the device tensor’s datum type and shape.

Trait Implementations

impl Debug for LazyHostStorage

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for LazyHostStorage

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for LazyHostStorage

impl PartialEq for LazyHostStorage

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl TensorStorage for LazyHostStorage

fn slice( &self, dt: DatumType, shape: &[usize], axis: usize, start: usize, end: usize, ) -> TractResult<Option<Tensor>>

Bytes still on the device are sliced there, and the slice stays lazy: an application truncating state it hands straight back to the next run() never brings either side across. Once they are here the generic copy is the cheaper path, and a host-side write has detached the device tensor, leaving nothing to ask.

fn byte_len(&self) -> usize

fn is_empty(&self) -> bool

fn deep_clone(&self) -> Box<dyn TensorStorage>

fn as_plain_ram(&self) -> Option<&PlainStorage>

fn as_plain_ram_mut(&mut self) -> Option<&mut PlainStorage>

fn into_plain_ram(self: Box<Self>) -> Option<PlainStorage>

fn dyn_hash(&self, _state: &mut dyn Hasher)

fn exotic_fact( &self, _shape: &[usize], ) -> TractResult<Option<Box<dyn ExoticFact>>>

Build the ExoticFact that describes this storage for use in TypedFact.

fn is_exotic(&self) -> bool

True when the tensor’s datum type and shape do not describe it on their own, so a fact over it carries an ExoticFact.

fn in_ram(&self) -> bool

True when the bytes are in host memory, readable without a transfer.

fn materialize_plain_ram(&self) -> TractResult<&PlainStorage>

Plain storage for the tensor’s bytes, producing it if this storage can.