Struct PaddedTokenBatch 

pub struct PaddedTokenBatch<B: Backend> {
    pub input_ids: Tensor<B, 2, Int>,
    pub attention_mask: Tensor<B, 2, Bool>,
    pub position_ids: Tensor<B, 2, Int>,
    pub lengths: Vec<usize>,
}

Available on crate feature std only.

Actual padded token inputs without labels, for a model’s encoder or decoder.

Fields

input_ids: Tensor<B, 2, Int>

[examples,maximum actual length], including explicitly masked padding cells.

attention_mask: Tensor<B, 2, Bool>

True exactly at actual input tokens, not a supervision/prompt mask.

position_ids: Tensor<B, 2, Int>

Actual per-document reset positions, with zero in masked padding cells.

lengths: Vec<usize>

Original lengths in actual paired example order.

Implementations

impl<B: Backend> PaddedTokenBatch<B>

pub fn examples(&self) -> usize

Actual document count, independent of rectangular token storage size.

pub fn validate(&self) -> Result<(), Seq2SeqDataError>

Check metadata geometry/device without downloading token or mask contents.

pub fn to_device(self, device: &B::Device) -> Self

Move actual input tensors without changing their geometry or host length metadata.

Trait Implementations

impl<B: Clone + Backend> Clone for PaddedTokenBatch<B>

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<B: Debug + Backend> Debug for PaddedTokenBatch<B>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.