Struct CausalBatcher 

pub struct CausalBatcher {
    pub layout: CausalBatchLayout,
    pub ignore_index: i64,
    pub target_alignment: CausalTargetAlignment,
    pub maximum_sequence_length: Option<usize>,
}

Available on crate feature std only.

Explicit supervision/layout policy shared by every call of a data batcher.

Fields

layout: CausalBatchLayout

Actual device token-axis geometry.

ignore_index: i64

Label sentinel, distinct from attention visibility and padding token ID.

target_alignment: CausalTargetAlignment

Must match the downstream loss’s causal-shift setting.

maximum_sequence_length: Option<usize>

Optional maximum actual sequence length; excess data is rejected, not cut.

Implementations

impl CausalBatcher

pub fn validate(&self, samples: &[CausalExample]) -> Result<(), CausalDataError>

Validate explicit policy and all actual source examples before uploading.

pub fn collate<B: Backend>( &self, samples: Vec<CausalExample>, device: &B::Device, ) -> Result<CausalBatch<B>, CausalDataError>

Collate explicit labels/IDs on the selected device, without tokenization.

Next-token mode ignores each document’s first target even for left padding, so a padded hidden state cannot supervise the first real token. Packed mode never inserts EOS, changes labels within a document, or authorizes a model to attend across the retained boundaries.

Trait Implementations

impl<B: Backend> Batcher<B, CausalExample, Result<CausalBatch<B>, CausalDataError>> for CausalBatcher

Available on crate feature dataset only.

fn batch( &self, items: Vec<CausalExample>, device: &B::Device, ) -> Result<CausalBatch<B>, CausalDataError>

Batches the given items on the specified device.

impl Clone for CausalBatcher

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CausalBatcher

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for CausalBatcher

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Eq for CausalBatcher

impl PartialEq for CausalBatcher

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for CausalBatcher

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for CausalBatcher