pub struct TokenMix { /* private fields */ }Expand description
Several corpora drawn from by weight, the way a pretraining mix of web text, code and books is: each sequence of a batch picks its source, then a window inside it.
Like TokenFile::batch, everything is drawn from the step number and
the mix’s own seed, so a resumed run sees the same batches. The seeds the
files were opened with are not used.
let mut mix = TokenMix::new(
vec![
(TokenFile::open("web.bin", 0)?, 0.7),
(TokenFile::open("code.bin", 0)?, 0.3),
],
42,
)?;
let batch = mix.batch(0, 8, 512)?;Implementations§
Source§impl TokenMix
impl TokenMix
Sourcepub fn new(
sources: Vec<(TokenFile, f64)>,
seed: u64,
) -> Result<Self, NetworkError>
pub fn new( sources: Vec<(TokenFile, f64)>, seed: u64, ) -> Result<Self, NetworkError>
Weights are relative and need not sum to one. A zero weight leaves a source out; a negative one, or none above zero, is an error.
Sourcepub fn batch(
&mut self,
step: u64,
sequences: usize,
seq_len: usize,
) -> Result<TokenBatch, NetworkError>
pub fn batch( &mut self, step: u64, sequences: usize, seq_len: usize, ) -> Result<TokenBatch, NetworkError>
sequences windows of seq_len tokens, drawn from step. See
TokenFile::batch.
Trait Implementations§
Auto Trait Implementations§
impl Forget for TokenMix
impl Freeze for TokenMix
impl Move for TokenMix
impl RefUnwindSafe for TokenMix
impl Send for TokenMix
impl Sync for TokenMix
impl Unpin for TokenMix
impl UnsafeUnpin for TokenMix
impl UnwindSafe for TokenMix
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more