pub struct MmapShardSource { /* private fields */ }Expand description
A CorpusRowSource over one or many shards with a bounded prefetch
window.
Implementations§
Source§impl MmapShardSource
impl MmapShardSource
Sourcepub fn open(paths: &[PathBuf]) -> Result<Self, ShardError>
pub fn open(paths: &[PathBuf]) -> Result<Self, ShardError>
Open a source over an explicit, ordered list of shard paths. The order
of paths defines the deterministic global row order, so the caller
must pass a stable ordering (e.g. lexicographically sorted filenames).
Sourcepub fn open_dir(dir: &Path) -> Result<Self, ShardError>
pub fn open_dir(dir: &Path) -> Result<Self, ShardError>
Open a source over every *.shard file in dir, ordered by file name
(the stable, OS-independent ordering that pins the deterministic global
row sequence).
Trait Implementations§
Source§impl CorpusRowSource for MmapShardSource
impl CorpusRowSource for MmapShardSource
Source§fn total_rows(&self) -> u64
fn total_rows(&self) -> u64
Total rows across every shard in this source.
Source§fn batch_rows(&self) -> usize
fn batch_rows(&self) -> usize
Rows handed back per
next_batch (may be smaller for the final batch).Source§fn next_batch(&mut self) -> Result<Option<RowBatch>, ShardError>
fn next_batch(&mut self) -> Result<Option<RowBatch>, ShardError>
Yield the next deterministic batch, or
Ok(None) at end of corpus.Auto Trait Implementations§
impl Freeze for MmapShardSource
impl RefUnwindSafe for MmapShardSource
impl Send for MmapShardSource
impl Sync for MmapShardSource
impl Unpin for MmapShardSource
impl UnsafeUnpin for MmapShardSource
impl UnwindSafe for MmapShardSource
Blanket Implementations§
impl<T> Allocation for T
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> DistributionExt for Twhere
T: ?Sized,
impl<T> DistributionExt for Twhere
T: ?Sized,
impl<T, U> Imply<T> for U
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> Pointable for T
impl<T> Pointable for T
impl<T> Read<Exclusive, BecauseExclusive> for Twhere
T: ?Sized,
Source§impl<SS, SP> SupersetOf<SS> for SPwhere
SS: SubsetOf<SP>,
impl<SS, SP> SupersetOf<SS> for SPwhere
SS: SubsetOf<SP>,
Source§fn to_subset(&self) -> Option<SS>
fn to_subset(&self) -> Option<SS>
The inverse inclusion map: attempts to construct
self from the equivalent element of its
superset. Read moreSource§fn is_in_subset(&self) -> bool
fn is_in_subset(&self) -> bool
Checks if
self is actually part of its subset T (and can be converted to it).Source§fn to_subset_unchecked(&self) -> SS
fn to_subset_unchecked(&self) -> SS
Use with care! Same as
self.to_subset but without any property checks. Always succeeds.Source§fn from_subset(element: &SS) -> SP
fn from_subset(element: &SS) -> SP
The inclusion map: converts
self to the equivalent element of its superset.