pub struct MemoryManagement<Storage> { /* private fields */ }Expand description
Reserves and keeps track of chunks of memory in the storage, and slices upon these chunks.
Implementations§
Source§impl<Storage: ComputeStorage> MemoryManagement<Storage>
impl<Storage: ComputeStorage> MemoryManagement<Storage>
Sourcepub fn from_configuration(
storage: Storage,
properties: &MemoryDeviceProperties,
config: MemoryConfiguration,
logger: Arc<ServerLogger>,
options: MemoryManagementOptions,
) -> Self
pub fn from_configuration( storage: Storage, properties: &MemoryDeviceProperties, config: MemoryConfiguration, logger: Arc<ServerLogger>, options: MemoryManagementOptions, ) -> Self
Creates the options from device limits.
Sourcepub fn mode(&mut self, mode: MemoryAllocationMode)
pub fn mode(&mut self, mode: MemoryAllocationMode)
Change the mode of allocation.
Sourcepub fn arena_begin(&mut self) -> u64
pub fn arena_begin(&mut self) -> u64
Route every subsequent reserve into the capture arena, and return the
generation this opening is.
The arena exists because a captured graph records ADDRESSES. Three things have to be true at once, and no ordinary pool gives all three:
- Reuse inside the region. A buffer born inside the captured region and dead inside it must hand its slice back, so the region’s peak footprint is its live set and not its allocation count. Without this every intra-region allocation becomes a driver allocation made while the stream is capturing, i.e. a graph MEMORY node.
- Whole-arena reservation. Once the region is captured, every slice the arena ever handed out is baked into a node, so NOTHING outside a capture may be given one for as long as the graph can be replayed. The arena is only ever consulted while it is open, and it never returns pages to the storage, so this holds for the whole arena and not just for the slices that happened to be live at capture end.
- A deterministic base. Reopening the arena resets no slice and frees nothing; slices are matched by exact effective size in a fixed order. A periodic region therefore issues an identical request sequence against an identical free set and gets identical addresses, which is what makes two captures of the same region comparable and a single graph legitimate for every step.
Note what this is NOT: it is not “hold the buffers a capture touched”. Holding gets reservation right and reuse wrong – it is precisely what stops (1) – and holding only the buffers that were live when the region opened gets reuse right and reservation wrong for everything born inside it. The two properties have to be separated onto different memory, which is what the arena is.
Buffers that were live when the arena opened are NOT in it. They belong to the ordinary pools, and if one dies inside the region its slice goes back to a pool the region is not allocating from – so it cannot be handed to a later node of the same region. That is the whole of the intra-region aliasing bug, closed structurally rather than by holding. The caller still owes such a buffer a hold for the graph’s lifetime, to stop the pool handing it out AFTER the capture closes.
TWO GRAPHS MAY SHARE ONE ARENA, and under (3) they positively should:
that is what makes their recorded addresses comparable, and it is what
lets a graph captured on one step stand in for the next. They then share
scratch, which is safe exactly when they are captures of the SAME region
and are replayed serially. Serial is a property of the one stream they
replay on; same-region is what Self::arena_signature is for.
Sourcepub fn arena_end(&mut self)
pub fn arena_end(&mut self)
Stop routing reserve into the arena. The arena keeps everything it
owns: closing it is the end of allocation from it, never the end of its
reservation.
Sourcepub fn arena_active(&self) -> bool
pub fn arena_active(&self) -> bool
Whether reserve is currently being served from the arena.
Sourcepub fn arena_stats(&self) -> ArenaStats
pub fn arena_stats(&self) -> ArenaStats
What the arena is holding, and how it got there.
Sourcepub fn arena_reset_counters(&mut self)
pub fn arena_reset_counters(&mut self)
Zero the served/miss counters, so a capture pass can be counted apart from the warm pass that preceded it. Does not free or reset anything.
Sourcepub fn arena_release(&mut self)
pub fn arena_release(&mut self)
Give the arena’s free pages back to the storage.
Only legal once no captured graph built against it can still be replayed – every such graph holds pointers into these pages. Slices still held by a live handle are kept.
Sourcepub fn arena_owns(&self, binding: &ManagedMemoryBinding) -> bool
pub fn arena_owns(&self, binding: &ManagedMemoryBinding) -> bool
Whether this binding’s memory came from the arena.
Sourcepub fn arena_mark(&self) -> usize
pub fn arena_mark(&self) -> usize
How many requests the open arena window has served so far. A caller brackets a region with two of these to name the window it owns.
Sourcepub fn arena_signature(&self, from: usize) -> u64
pub fn arena_signature(&self, from: usize) -> u64
A hash of every size served since from, in order: the signature of the
region that allocated them. See Self::arena_sizes.
Sourcepub fn cleanup(&mut self, explicit: bool)
pub fn cleanup(&mut self, explicit: bool)
Cleanup allocations in pools that are deemed unnecessary.
Sourcepub fn get_cursor(&self, binding: ManagedMemoryBinding) -> Result<u64, IoError>
pub fn get_cursor(&self, binding: ManagedMemoryBinding) -> Result<u64, IoError>
Returns the storage from the specified binding
Sourcepub fn get_storage(
&mut self,
binding: ManagedMemoryBinding,
) -> Result<StorageHandle, IoError>
pub fn get_storage( &mut self, binding: ManagedMemoryBinding, ) -> Result<StorageHandle, IoError>
Returns the storage from the specified binding
Sourcepub fn register_external(
&mut self,
storage: StorageHandle,
) -> ManagedMemoryHandle
pub fn register_external( &mut self, storage: StorageHandle, ) -> ManagedMemoryHandle
Register an externally-created StorageHandle (e.g. a GPU buffer that
aliases mmap’d host memory) as a tensor-addressable memory handle,
BYPASSING the allocation pools. The returned handle resolves to storage
via Self::get_storage/Self::get_resource but is never reserved,
reused, sub-sliced, or reclaimed — the caller owns the underlying storage
lifetime. This is the zero-copy seam for weights-as-tribles: an mmap’d f16
pile page becomes a GPU tensor with no copy and no allocation.
The returned handle (and every clone of it) permanently reports
can_mut() == false: external storage aliases READ-ONLY host memory,
so it must never be picked as an in-place kernel destination. Burn’s
elementwise kernels reuse an “owned” input handle as the output buffer
(can_mut() = a handle-count heuristic, strong_count <= 2); on a
read-only mmap the GPU’s in-place write is silently dropped by the
pager and the op returns its INPUT bytes — e.g. voxtral’s folded-ear
load saw q.mul_scalar(1/sqrt(d)) come back unscaled (2026-07-12,
caught by the aliased-vs-materialized A/B being token-divergent).
Pinning two extra handle clones for the registration’s life (external
entries are never removed anyway) pushes every user-facing clone over
the heuristic’s threshold, forcing all consumers to allocate outputs.
Sourcepub fn get_resource(
&mut self,
binding: ManagedMemoryBinding,
offset_start: Option<u64>,
offset_end: Option<u64>,
) -> Result<Storage::Resource, IoError>
pub fn get_resource( &mut self, binding: ManagedMemoryBinding, offset_start: Option<u64>, offset_end: Option<u64>, ) -> Result<Storage::Resource, IoError>
Returns the resource from the storage at the specified handle
Sourcepub fn reserve(&mut self, size: u64) -> Result<ManagedMemoryHandle, IoError>
pub fn reserve(&mut self, size: u64) -> Result<ManagedMemoryHandle, IoError>
Finds a spot in memory for a resource with the given size in bytes, and returns a handle to it
Sourcepub fn reserve_timing(&self) -> (u64, u64)
pub fn reserve_timing(&self) -> (u64, u64)
What a reservation costs, as (calls, nanoseconds) since the last
reset. Zero unless CUBECL_TIME_RESERVE=1.
Sourcepub fn reserve_timing_reset(&mut self)
pub fn reserve_timing_reset(&mut self)
Zero the reservation timer, so one pass can be counted apart from the startup that preceded it.
Sourcepub fn storage(&mut self) -> &mut Storage
pub fn storage(&mut self) -> &mut Storage
Fetch the storage used by the memory manager.
§Notes
The storage should probably not be used for allocations since the handles won’t be compatible with the ones provided by the current trait. Prefer using the alloc and dealloc functions.
This is useful if you need to time the deallocations based on async computation, or to change the mode of storage for different reasons.
Sourcepub fn memory_usage(&self) -> MemoryUsage
pub fn memory_usage(&self) -> MemoryUsage
Get the current memory usage.
Sourcepub fn print_memory_usage(&self)
pub fn print_memory_usage(&self)
Print out a report of the current memory usage.
Sourcepub fn bind(
&mut self,
reserved: ManagedMemoryHandle,
assigned: ManagedMemoryHandle,
cursor: u64,
) -> Result<(), IoError>
pub fn bind( &mut self, reserved: ManagedMemoryHandle, assigned: ManagedMemoryHandle, cursor: u64, ) -> Result<(), IoError>
Binds the given handle to a [MemorySlot].