pub struct StreamingIngester<'a> { /* private fields */ }Expand description
Streaming ingester for memory-efficient file encoding
Processes data in chunks as it arrives, maintaining bounded memory usage. Suitable for files of any size including multi-GB datasets.
Implementations§
Source§impl<'a> StreamingIngester<'a>
impl<'a> StreamingIngester<'a>
Sourcepub fn builder(fs: &'a VersionedEmbrFS) -> StreamingIngesterBuilder<'a>
pub fn builder(fs: &'a VersionedEmbrFS) -> StreamingIngesterBuilder<'a>
Create a builder for configuring a streaming ingester
Sourcepub fn ingest_reader<R: Read>(&mut self, reader: R) -> Result<(), EmbrFSError>
pub fn ingest_reader<R: Read>(&mut self, reader: R) -> Result<(), EmbrFSError>
Ingest data from a reader
Reads data in chunks and encodes progressively. Memory usage is bounded
by chunk_size regardless of total file size.
Sourcepub fn ingest_buffered<R: BufRead>(
&mut self,
reader: R,
) -> Result<(), EmbrFSError>
pub fn ingest_buffered<R: BufRead>( &mut self, reader: R, ) -> Result<(), EmbrFSError>
Ingest data from a buffered reader (more efficient)
Sourcepub fn ingest_bytes(&mut self, data: &[u8]) -> Result<(), EmbrFSError>
pub fn ingest_bytes(&mut self, data: &[u8]) -> Result<(), EmbrFSError>
Ingest a slice of bytes
Can be called multiple times. Data is buffered until a full chunk is available, then encoded and stored.
Sourcepub fn finalize(self) -> Result<StreamingResult, EmbrFSError>
pub fn finalize(self) -> Result<StreamingResult, EmbrFSError>
Finalize the ingestion and commit to filesystem
Processes any remaining buffered data and commits all chunks to the versioned filesystem atomically.
Sourcepub fn progress(&self) -> StreamingProgress
pub fn progress(&self) -> StreamingProgress
Get current progress
Auto Trait Implementations§
impl<'a> !Freeze for StreamingIngester<'a>
impl<'a> RefUnwindSafe for StreamingIngester<'a>
impl<'a> Send for StreamingIngester<'a>
impl<'a> Sync for StreamingIngester<'a>
impl<'a> Unpin for StreamingIngester<'a>
impl<'a> UnwindSafe for StreamingIngester<'a>
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more