pub struct StreamingLoader { /* private fields */ }Expand description
Streams a GGUF model file layer-by-layer without loading it all into RAM.
Implementations§
Source§impl StreamingLoader
impl StreamingLoader
pub fn new(path: &Path, num_layers: usize) -> Self
pub fn verbose(self) -> Self
Sourcepub fn should_stream(model_size_mb: f64, vram_free_mb: u64) -> bool
pub fn should_stream(model_size_mb: f64, vram_free_mb: u64) -> bool
Returns true when streaming should be used.
model_size_mb: total model weight size in MiB.
vram_free_mb: free VRAM reported by the hardware profiler.
Sourcepub fn for_each_layer<F>(&self, f: F) -> Result<()>
pub fn for_each_layer<F>(&self, f: F) -> Result<()>
Iterate over all layers, calling f for each.
The mmap is held open for the duration of the callback to allow
zero-copy access to raw bytes. In production f would quantize the
layer weights and write the compressed result to the output .tptf.
Auto Trait Implementations§
impl Freeze for StreamingLoader
impl RefUnwindSafe for StreamingLoader
impl Send for StreamingLoader
impl Sync for StreamingLoader
impl Unpin for StreamingLoader
impl UnsafeUnpin for StreamingLoader
impl UnwindSafe for StreamingLoader
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more