Struct FcdStream 

pub struct FcdStream<'a> {
    pub ids: &'a [u32],
    pub weight: Option<&'a [f32]>,
    pub starts: Option<&'a [usize]>,
}

A training or validation token stream for the polish.

weight[i] scales the loss of predicting ids[i] (0 = context only), so an answer-only loss is a stream whose prompt tokens weigh 0. starts marks record boundaries (ascending, starts[0] == 0): a window then begins inside one record and nothing past that record’s end is scored, so one prompt’s answer is never trained on another’s context. Without either, the stream is plain text and every position counts — exactly the original corpus behaviour.

Fields

ids: &'a [u32]

weight: Option<&'a [f32]>

starts: Option<&'a [usize]>

Implementations

impl<'a> FcdStream<'a>

pub fn plain(ids: &'a [u32]) -> Self

Trait Implementations

impl<'a> Clone for FcdStream<'a>

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Copy for FcdStream<'a>

impl<'a> Debug for FcdStream<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.