Struct BatchInputs 

pub struct BatchInputs<T> {
    pub input_ids: T,
    pub attention_mask: T,
    pub token_type_ids: T,
    pub position_ids: T,
    pub float_mask: T,
    pub pooling_mask: T,
    pub batch: usize,
    pub max_seq: usize,
    pub total_tokens: usize,
    pub seq_lengths: Vec<usize>,
    pub cu_seqlens: Option<Vec<usize>>,
}

Batch input tensors on device, produced by Driver::prepare_batch.

Supports both padded and unpadded modes:

• Padded: all sequences padded to max_seq. cu_seqlens is None.
• Unpadded: sequences concatenated without padding. cu_seqlens contains cumulative lengths [0, len0, len0+len1, ...] so attention knows where each sequence starts. Eliminates ALL padding compute.

Fields

input_ids: T

Token IDs — [batch * max_seq] (padded) or [total_tokens] (unpadded).

attention_mask: T

Attention mask [batch * max_seq] as int32 (0 or 1). Unused in unpadded mode.

token_type_ids: T

Token type IDs — same layout as input_ids.

position_ids: T

Position IDs — same layout as input_ids.

float_mask: T

Float attention bias mask [batch * max_seq] (0.0 or -1e9) for softmax.

pooling_mask: T

Float pooling mask [batch * max_seq] (1.0 or 0.0) for mean pooling.

batch: usize

Number of sequences in this batch.

max_seq: usize

Maximum sequence length (all sequences padded to this). In unpadded mode, this is the longest sequence (used for workspace sizing, not padding).

total_tokens: usize

Total actual tokens across all sequences (no padding).

seq_lengths: Vec<usize>

Per-sequence lengths: [batch] — each element is the actual token count.

cu_seqlens: Option<Vec<usize>>

Cumulative sequence lengths for unpadded attention: [batch + 1]. cu_seqlens[i]..cu_seqlens[i+1] is the token range for sequence i. None in padded mode (all sequences padded to max_seq).