Skip to main content

MemoryManagement

Struct MemoryManagement 

Source
pub struct MemoryManagement<Storage> { /* private fields */ }
Expand description

Reserves and keeps track of chunks of memory in the storage, and slices upon these chunks.

Implementations§

Source§

impl<Storage: ComputeStorage> MemoryManagement<Storage>

Source

pub fn from_configuration( storage: Storage, properties: &MemoryDeviceProperties, config: MemoryConfiguration, logger: Arc<ServerLogger>, options: MemoryManagementOptions, ) -> Self

Creates the options from device limits.

Source

pub fn mode(&mut self, mode: MemoryAllocationMode)

Change the mode of allocation.

Source

pub fn arena_begin(&mut self) -> u64

Route every subsequent reserve into the capture arena, and return the generation this opening is.

The arena exists because a captured graph records ADDRESSES. Three things have to be true at once, and no ordinary pool gives all three:

  1. Reuse inside the region. A buffer born inside the captured region and dead inside it must hand its slice back, so the region’s peak footprint is its live set and not its allocation count. Without this every intra-region allocation becomes a driver allocation made while the stream is capturing, i.e. a graph MEMORY node.
  2. Whole-arena reservation. Once the region is captured, every slice the arena ever handed out is baked into a node, so NOTHING outside a capture may be given one for as long as the graph can be replayed. The arena is only ever consulted while it is open, and it never returns pages to the storage, so this holds for the whole arena and not just for the slices that happened to be live at capture end.
  3. A deterministic base. Reopening the arena resets no slice and frees nothing; slices are matched by exact effective size in a fixed order. A periodic region therefore issues an identical request sequence against an identical free set and gets identical addresses, which is what makes two captures of the same region comparable and a single graph legitimate for every step.

Note what this is NOT: it is not “hold the buffers a capture touched”. Holding gets reservation right and reuse wrong – it is precisely what stops (1) – and holding only the buffers that were live when the region opened gets reuse right and reservation wrong for everything born inside it. The two properties have to be separated onto different memory, which is what the arena is.

Buffers that were live when the arena opened are NOT in it. They belong to the ordinary pools, and if one dies inside the region its slice goes back to a pool the region is not allocating from – so it cannot be handed to a later node of the same region. That is the whole of the intra-region aliasing bug, closed structurally rather than by holding. The caller still owes such a buffer a hold for the graph’s lifetime, to stop the pool handing it out AFTER the capture closes.

TWO GRAPHS MAY SHARE ONE ARENA, and under (3) they positively should: that is what makes their recorded addresses comparable, and it is what lets a graph captured on one step stand in for the next. They then share scratch, which is safe exactly when they are captures of the SAME region and are replayed serially. Serial is a property of the one stream they replay on; same-region is what Self::arena_signature is for.

Source

pub fn arena_end(&mut self)

Stop routing reserve into the arena. The arena keeps everything it owns: closing it is the end of allocation from it, never the end of its reservation.

Source

pub fn arena_active(&self) -> bool

Whether reserve is currently being served from the arena.

Source

pub fn arena_stats(&self) -> ArenaStats

What the arena is holding, and how it got there.

Source

pub fn arena_reset_counters(&mut self)

Zero the served/miss counters, so a capture pass can be counted apart from the warm pass that preceded it. Does not free or reset anything.

Source

pub fn arena_release(&mut self)

Give the arena’s free pages back to the storage.

Only legal once no captured graph built against it can still be replayed – every such graph holds pointers into these pages. Slices still held by a live handle are kept.

Source

pub fn arena_owns(&self, binding: &ManagedMemoryBinding) -> bool

Whether this binding’s memory came from the arena.

Source

pub fn arena_mark(&self) -> usize

How many requests the open arena window has served so far. A caller brackets a region with two of these to name the window it owns.

Source

pub fn arena_signature(&self, from: usize) -> u64

A hash of every size served since from, in order: the signature of the region that allocated them. See Self::arena_sizes.

Source

pub fn cleanup(&mut self, explicit: bool)

Cleanup allocations in pools that are deemed unnecessary.

Source

pub fn get_cursor(&self, binding: ManagedMemoryBinding) -> Result<u64, IoError>

Returns the storage from the specified binding

Source

pub fn get_storage( &mut self, binding: ManagedMemoryBinding, ) -> Result<StorageHandle, IoError>

Returns the storage from the specified binding

Source

pub fn register_external( &mut self, storage: StorageHandle, ) -> ManagedMemoryHandle

Register an externally-created StorageHandle (e.g. a GPU buffer that aliases mmap’d host memory) as a tensor-addressable memory handle, BYPASSING the allocation pools. The returned handle resolves to storage via Self::get_storage/Self::get_resource but is never reserved, reused, sub-sliced, or reclaimed — the caller owns the underlying storage lifetime. This is the zero-copy seam for weights-as-tribles: an mmap’d f16 pile page becomes a GPU tensor with no copy and no allocation.

The returned handle (and every clone of it) permanently reports can_mut() == false: external storage aliases READ-ONLY host memory, so it must never be picked as an in-place kernel destination. Burn’s elementwise kernels reuse an “owned” input handle as the output buffer (can_mut() = a handle-count heuristic, strong_count <= 2); on a read-only mmap the GPU’s in-place write is silently dropped by the pager and the op returns its INPUT bytes — e.g. voxtral’s folded-ear load saw q.mul_scalar(1/sqrt(d)) come back unscaled (2026-07-12, caught by the aliased-vs-materialized A/B being token-divergent). Pinning two extra handle clones for the registration’s life (external entries are never removed anyway) pushes every user-facing clone over the heuristic’s threshold, forcing all consumers to allocate outputs.

Source

pub fn get_resource( &mut self, binding: ManagedMemoryBinding, offset_start: Option<u64>, offset_end: Option<u64>, ) -> Result<Storage::Resource, IoError>

Returns the resource from the storage at the specified handle

Source

pub fn reserve(&mut self, size: u64) -> Result<ManagedMemoryHandle, IoError>

Finds a spot in memory for a resource with the given size in bytes, and returns a handle to it

Source

pub fn reserve_timing(&self) -> (u64, u64)

What a reservation costs, as (calls, nanoseconds) since the last reset. Zero unless CUBECL_TIME_RESERVE=1.

Source

pub fn reserve_timing_reset(&mut self)

Zero the reservation timer, so one pass can be counted apart from the startup that preceded it.

Source

pub fn storage(&mut self) -> &mut Storage

Fetch the storage used by the memory manager.

§Notes

The storage should probably not be used for allocations since the handles won’t be compatible with the ones provided by the current trait. Prefer using the alloc and dealloc functions.

This is useful if you need to time the deallocations based on async computation, or to change the mode of storage for different reasons.

Source

pub fn memory_usage(&self) -> MemoryUsage

Get the current memory usage.

Source

pub fn print_memory_usage(&self)

Print out a report of the current memory usage.

Source

pub fn bind( &mut self, reserved: ManagedMemoryHandle, assigned: ManagedMemoryHandle, cursor: u64, ) -> Result<(), IoError>

Binds the given handle to a [MemorySlot].

Trait Implementations§

Source§

impl<Storage> Debug for MemoryManagement<Storage>

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl<Storage: ComputeStorage> Display for MemoryManagement<Storage>

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more

Auto Trait Implementations§

§

impl<Storage> !RefUnwindSafe for MemoryManagement<Storage>

§

impl<Storage> !UnwindSafe for MemoryManagement<Storage>

§

impl<Storage> Freeze for MemoryManagement<Storage>
where Storage: Freeze,

§

impl<Storage> Send for MemoryManagement<Storage>
where Storage: Send,

§

impl<Storage> Sync for MemoryManagement<Storage>
where Storage: Sync,

§

impl<Storage> Unpin for MemoryManagement<Storage>
where Storage: Unpin,

§

impl<Storage> UnsafeUnpin for MemoryManagement<Storage>
where Storage: UnsafeUnpin,

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T> ToString for T
where T: Display + ?Sized,

Source§

fn to_string(&self) -> String

Converts the given value to a String. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.