Struct PackedTokenBatch 

pub struct PackedTokenBatch<B: Backend> {
    pub input_ids: Tensor<B, 1, Int>,
    pub attention_mask: Tensor<B, 1, Bool>,
    pub position_ids: Tensor<B, 1, Int>,
    pub boundaries: Vec<usize>,
}

Available on crate feature std only.

Actual flat document inputs without labels or synthetic separator tokens.

Fields

input_ids: Tensor<B, 1, Int>

Flat actual tokenizer IDs.

attention_mask: Tensor<B, 1, Bool>

Actual input visibility; ordinary packed collation marks every real token visible.

position_ids: Tensor<B, 1, Int>

Actual per-document positions reset to zero.

boundaries: Vec<usize>

Cumulative source/decoder boundaries, independently beginning at zero.

Implementations

impl<B: Backend> PackedTokenBatch<B>

pub fn examples(&self) -> usize

Actual retained document count, including any explicitly empty sources.

pub fn validate(&self) -> Result<(), Seq2SeqDataError>

Check exact physical boundaries/geometry/device without tensor readback.

pub fn to_device(self, device: &B::Device) -> Self

Move actual flat inputs while retaining their exact document boundaries.

Trait Implementations

impl<B: Clone + Backend> Clone for PackedTokenBatch<B>

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<B: Debug + Backend> Debug for PackedTokenBatch<B>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.