Struct BucketedCompileCache 

pub struct BucketedCompileCache { /* private fields */ }

Implementations

impl BucketedCompileCache

pub fn new(device: Device, buckets: Vec<Range<u64>>) -> Self

pub fn power_of_two_ladder(device: Device, min: u64, max: u64) -> Self

Power-of-two ladder over [1, max], with extents [min_pow2, 2·min_pow2, 4·min_pow2, …, max_pow2] where min_pow2 = min.next_power_of_two() and max_pow2 is the smallest power of two ≥ max. Each bucket compiles at its upper-bound extent, so an actual value in bucket (prev_extent .. ext] runs kernels at extent ext (not at the worst case of the whole range). Guarantees compute waste from padding ≤2× — actual > ext / 2 for every bucket except possibly the smallest.

Example: power_of_two_ladder(Device::Cpu, 8, 256) yields buckets 1..9, 9..17, 17..33, 33..65, 65..129, 129..257 with compile extents 8, 16, 32, 64, 128, 256. An actual = 17 runs at extent 32 instead of the 255 a single wide 1..256 bucket would compile at — that’s the “skip compute” win, paid for with O(log max) compiled artifacts instead of one.

pub fn power_of_two_ladder_with_policy( device: Device, min: u64, max: u64, policy: Option<PrecisionPolicy>, ) -> Self

pub fn with_policy( device: Device, buckets: Vec<Range<u64>>, policy: Option<PrecisionPolicy>, ) -> Self

pub fn get_or_compile<F: FnOnce(u64) -> Graph>( &mut self, key: u64, build: F, ) -> Option<(u64, &mut CompiledGraph)>

Find the bucket containing key, compile if needed, return (upper, &mut CompiledGraph) where upper = range.end - 1 is the extent the graph was compiled for. Caller pads inputs to upper before calling run. Returns None if key is outside every bucket — caller decides whether to fall back to a one-off compile.

build receives upper and must return a Graph specialized for that extent.

pub fn get_or_compile_with_options<F: FnOnce(u64) -> Graph>( &mut self, key: u64, build: F, options: &CompileOptions, ) -> Option<(u64, &mut CompiledGraph)>

Like Self::get_or_compile with explicit [CompileOptions].

pub fn get_or_compile_hir<F: FnOnce(u64) -> HirModule>( &mut self, key: u64, build: F, ) -> Option<(u64, &mut CompiledGraph)>

Like Self::get_or_compile but builds and compiles HIR directly through the fusion-first pipeline (Session::compile_hir).

pub fn get_or_compile_hir_with_options<F: FnOnce(u64) -> HirModule>( &mut self, key: u64, build: F, options: &CompileOptions, ) -> Option<(u64, &mut CompiledGraph)>

Like Self::get_or_compile_hir with explicit [CompileOptions] (tier-1 profile, fusion target, …).

pub fn bucket_for(&self, key: u64) -> Option<usize>

Index of the bucket containing key, or None if out of range. Linear scan — bucket counts are small in practice.

pub fn buckets(&self) -> impl Iterator<Item = &Range<u64>>

pub fn compiled_count(&self) -> usize

Number of buckets that have been compiled so far (≤ total buckets).

pub fn total_buckets(&self) -> usize

pub fn run_padded<F: FnOnce(u64) -> Graph>( &mut self, key: u64, actual_rows: usize, build: F, inputs: &[(&str, &[f32], usize)], output_inners: &[usize], ) -> Option<(u64, Vec<Vec<f32>>)>

“Compile at max, run at less” convenience for inputs and outputs whose outer dimension is the bucket key:

1. Find or compile the bucket containing key.
2. For each input, pad to upper rows along the outer dim using pad_rows (caller passes the inner-dim stride per input; inner = 1 for purely 1D inputs).
3. Run the compiled graph at full extent.
4. Slice each output back to actual_rows along its outer dim. Outputs flagged with inner = 0 in output_inners are returned unsliced (use this for extent-independent outputs like a pooled [hidden] embedding). Missing entries past the end of output_inners are also returned unsliced.

Returns (upper, outputs). Returns None if key falls outside every bucket.

Compute scope: kernels execute at the bucket’s compile extent (upper), not at actual_rows. This means smaller buckets directly translate to less padded compute. With power_of_two_ladder the worst- case waste is bounded at 2×; with hand-tuned buckets it can be arbitrarily tight. True active-extent dispatch — one big compile, kernels short-circuit at runtime — is a separate per-backend change.

pub fn ensure_graph_with_params<F>( &mut self, key: u64, build: F, options: &CompileOptions, ) -> Option<(u64, &mut CompiledGraph)>

where F: FnOnce(u64) -> (Graph, HashMap<String, Vec<f32>>),

Like Self::get_or_compile_with_options but also uploads params on first compile.

pub fn ensure_hir_with_params<F>( &mut self, key: u64, build: F, options: &CompileOptions, ) -> Option<(u64, &mut CompiledGraph)>

where F: FnOnce(u64) -> (HirModule, HashMap<String, Vec<f32>>),

HIR variant of Self::ensure_graph_with_params.

pub fn run_padded_mixed<F>( &mut self, key: u64, actual_rows: usize, build: F, inputs: &[CacheRunInput<'_>], output_inners: &[usize], ) -> Option<(u64, Vec<Vec<f32>>)>

where F: FnOnce(u64) -> Graph,

Self::run_padded with per-input optional row padding (CacheRunInput).