Struct CausalExample 

pub struct CausalExample {
    pub input_ids: Vec<i64>,
    pub labels: Vec<i64>,
}

Available on crate feature std only.

One already-encoded document with caller-declared supervision.

Fields

input_ids: Vec<i64>

Nonnegative actual tokenizer IDs, without batch-padding tokens.

labels: Vec<i64>

Explicit token targets or the collator’s selected ignore_index.

Implementations

impl CausalExample

pub fn validate(&self, ignore_index: i64) -> Result<(), CausalDataError>

Validate a nonempty encoded document without guessing prompt spans.

pub fn length(&self) -> usize

Actual encoded sequence length, excluding any future batch padding.

Trait Implementations

impl<B: Backend> Batcher<B, CausalExample, Result<CausalBatch<B>, CausalDataError>> for CausalBatcher

Available on crate feature dataset only.

fn batch( &self, items: Vec<CausalExample>, device: &B::Device, ) -> Result<CausalBatch<B>, CausalDataError>

Batches the given items on the specified device.

impl Clone for CausalExample

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CausalExample

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for CausalExample

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Eq for CausalExample

impl PartialEq for CausalExample

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for CausalExample

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for CausalExample