pub struct StepProfiler { /* private fields */ }Expand description
Per-step timing accumulator.
Usage: call begin(phase) before each section, end(phase) after.
Call finish_step() to record the step and optionally print a report.
Per-layer profiling (PMAT-480): call begin_layer(layer) / end_layer_fwd(layer)
and end_layer_bwd(layer) inside the forward/backward loops to capture
per-layer timing. Reports per-layer breakdown when enabled.
Implementations§
Source§impl StepProfiler
impl StepProfiler
pub const EMBED: usize = EMBED
pub const OP_RMSNORM_ATTN: usize = OP_RMSNORM_ATTN
pub const OP_QKV_GEMM: usize = OP_QKV_GEMM
pub const OP_ATTENTION: usize = OP_ATTENTION
pub const OP_O_PROJ: usize = OP_O_PROJ
pub const OP_RMSNORM_FFN: usize = OP_RMSNORM_FFN
pub const OP_GATE_UP_GEMM: usize = OP_GATE_UP_GEMM
pub const OP_SILU: usize = OP_SILU
pub const OP_DOWN_GEMM: usize = OP_DOWN_GEMM
pub const OP_LORA: usize = OP_LORA
pub const OP_DOWN_BWD: usize = OP_DOWN_BWD
pub const OP_SWIGLU_BWD: usize = OP_SWIGLU_BWD
pub const OP_GATE_UP_BWD: usize = OP_GATE_UP_BWD
pub const OP_ATTN_BWD: usize = OP_ATTN_BWD
pub const OP_QKV_BWD: usize = OP_QKV_BWD
pub const OP_NORM_BWD: usize = OP_NORM_BWD
pub const OP_LORA_BWD: usize = OP_LORA_BWD
pub const H2D: usize = H2D
pub const FORWARD: usize = FORWARD
pub const NORM_LM: usize = NORM_LM
pub const LOSS: usize = LOSS
pub const GRAD_H2D: usize = GRAD_H2D
pub const LM_BWD: usize = LM_BWD
pub const NORM_BWD: usize = NORM_BWD
pub const BLK_BWD: usize = BLK_BWD
pub const EMBED_BWD: usize = EMBED_BWD
pub const OPT: usize = OPT
Sourcepub fn new(enabled: bool, report_interval: usize) -> Self
pub fn new(enabled: bool, report_interval: usize) -> Self
Create a new profiler.
enabled: if false, all methods are no-opsreport_interval: print summary every N steps (0 = manual only)
Sourcepub fn begin_step(&mut self)
pub fn begin_step(&mut self)
Mark the start of a training step.
Sourcepub fn begin(&mut self, _phase: usize)
pub fn begin(&mut self, _phase: usize)
Mark the start of a phase. Must call end(phase) to record.
Sourcepub fn begin_layer(&mut self)
pub fn begin_layer(&mut self)
Start per-layer timing (PMAT-480). Call before each layer’s forward or backward.
Sourcepub fn end_layer_fwd(&mut self, layer: usize)
pub fn end_layer_fwd(&mut self, layer: usize)
Record per-layer forward time (PMAT-480). Call after layer forward completes.
Sourcepub fn end_layer_bwd(&mut self, layer: usize)
pub fn end_layer_bwd(&mut self, layer: usize)
Record per-layer backward time (PMAT-480). Call after layer backward completes.
Sourcepub fn finish_step(&mut self)
pub fn finish_step(&mut self)
Finish the current step. Records totals and optionally prints report.
Sourcepub fn print_report(&self)
pub fn print_report(&self)
Print cumulative profiling report to stdout.
Sourcepub fn begin_op(&mut self)
pub fn begin_op(&mut self)
PMAT-483/entrenar#328: Start timing a per-operation phase within a layer.
Sourcepub fn end_op(&mut self, op: usize)
pub fn end_op(&mut self, op: usize)
PMAT-483/entrenar#328: Record elapsed time for a per-operation phase.
Sourcepub fn end_op_raw(&mut self, op: usize, us: u64)
pub fn end_op_raw(&mut self, op: usize, us: u64)
PMAT-483: Feed pre-accumulated microseconds for an operation (from CudaBlockScratch).
Sourcepub fn is_enabled(&self) -> bool
pub fn is_enabled(&self) -> bool
Whether the profiler is active.
Sourcepub fn step_count(&self) -> usize
pub fn step_count(&self) -> usize
Get step count.
Sourcepub fn print_json_report(&self)
pub fn print_json_report(&self)
PMAT-483: Emit structured JSON profiling report to stderr. Parseable by canary scripts for scientific analysis.
Sourcepub fn record_layer_times(&mut self, fwd_us: &[u64], bwd_us: &[u64])
pub fn record_layer_times(&mut self, fwd_us: &[u64], bwd_us: &[u64])
PMAT-483: Feed per-layer timing data from InstructGpuTrainingState. Call once per step after forward+backward complete.
Auto Trait Implementations§
impl Freeze for StepProfiler
impl RefUnwindSafe for StepProfiler
impl Send for StepProfiler
impl Sync for StepProfiler
impl Unpin for StepProfiler
impl UnsafeUnpin for StepProfiler
impl UnwindSafe for StepProfiler
Blanket Implementations§
impl<T> Allocation for T
impl<T> Allocation for T
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> FmtForward for T
impl<T> FmtForward for T
Source§fn fmt_binary(self) -> FmtBinary<Self>where
Self: Binary,
fn fmt_binary(self) -> FmtBinary<Self>where
Self: Binary,
self to use its Binary implementation when Debug-formatted.Source§fn fmt_display(self) -> FmtDisplay<Self>where
Self: Display,
fn fmt_display(self) -> FmtDisplay<Self>where
Self: Display,
self to use its Display implementation when
Debug-formatted.Source§fn fmt_lower_exp(self) -> FmtLowerExp<Self>where
Self: LowerExp,
fn fmt_lower_exp(self) -> FmtLowerExp<Self>where
Self: LowerExp,
self to use its LowerExp implementation when
Debug-formatted.Source§fn fmt_lower_hex(self) -> FmtLowerHex<Self>where
Self: LowerHex,
fn fmt_lower_hex(self) -> FmtLowerHex<Self>where
Self: LowerHex,
self to use its LowerHex implementation when
Debug-formatted.Source§fn fmt_octal(self) -> FmtOctal<Self>where
Self: Octal,
fn fmt_octal(self) -> FmtOctal<Self>where
Self: Octal,
self to use its Octal implementation when Debug-formatted.Source§fn fmt_pointer(self) -> FmtPointer<Self>where
Self: Pointer,
fn fmt_pointer(self) -> FmtPointer<Self>where
Self: Pointer,
self to use its Pointer implementation when
Debug-formatted.Source§fn fmt_upper_exp(self) -> FmtUpperExp<Self>where
Self: UpperExp,
fn fmt_upper_exp(self) -> FmtUpperExp<Self>where
Self: UpperExp,
self to use its UpperExp implementation when
Debug-formatted.Source§fn fmt_upper_hex(self) -> FmtUpperHex<Self>where
Self: UpperHex,
fn fmt_upper_hex(self) -> FmtUpperHex<Self>where
Self: UpperHex,
self to use its UpperHex implementation when
Debug-formatted.Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
Source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> Pipe for Twhere
T: ?Sized,
impl<T> Pipe for Twhere
T: ?Sized,
Source§fn pipe<R>(self, func: impl FnOnce(Self) -> R) -> Rwhere
Self: Sized,
fn pipe<R>(self, func: impl FnOnce(Self) -> R) -> Rwhere
Self: Sized,
Source§fn pipe_ref<'a, R>(&'a self, func: impl FnOnce(&'a Self) -> R) -> Rwhere
R: 'a,
fn pipe_ref<'a, R>(&'a self, func: impl FnOnce(&'a Self) -> R) -> Rwhere
R: 'a,
self and passes that borrow into the pipe function. Read moreSource§fn pipe_ref_mut<'a, R>(&'a mut self, func: impl FnOnce(&'a mut Self) -> R) -> Rwhere
R: 'a,
fn pipe_ref_mut<'a, R>(&'a mut self, func: impl FnOnce(&'a mut Self) -> R) -> Rwhere
R: 'a,
self and passes that borrow into the pipe function. Read moreSource§fn pipe_borrow<'a, B, R>(&'a self, func: impl FnOnce(&'a B) -> R) -> R
fn pipe_borrow<'a, B, R>(&'a self, func: impl FnOnce(&'a B) -> R) -> R
Source§fn pipe_borrow_mut<'a, B, R>(
&'a mut self,
func: impl FnOnce(&'a mut B) -> R,
) -> R
fn pipe_borrow_mut<'a, B, R>( &'a mut self, func: impl FnOnce(&'a mut B) -> R, ) -> R
Source§fn pipe_as_ref<'a, U, R>(&'a self, func: impl FnOnce(&'a U) -> R) -> R
fn pipe_as_ref<'a, U, R>(&'a self, func: impl FnOnce(&'a U) -> R) -> R
self, then passes self.as_ref() into the pipe function.Source§fn pipe_as_mut<'a, U, R>(&'a mut self, func: impl FnOnce(&'a mut U) -> R) -> R
fn pipe_as_mut<'a, U, R>(&'a mut self, func: impl FnOnce(&'a mut U) -> R) -> R
self, then passes self.as_mut() into the pipe
function.Source§fn pipe_deref<'a, T, R>(&'a self, func: impl FnOnce(&'a T) -> R) -> R
fn pipe_deref<'a, T, R>(&'a self, func: impl FnOnce(&'a T) -> R) -> R
self, then passes self.deref() into the pipe function.Source§impl<T> Pointable for T
impl<T> Pointable for T
Source§impl<T> PolicyExt for Twhere
T: ?Sized,
impl<T> PolicyExt for Twhere
T: ?Sized,
Source§impl<T> Tap for T
impl<T> Tap for T
Source§fn tap_borrow<B>(self, func: impl FnOnce(&B)) -> Self
fn tap_borrow<B>(self, func: impl FnOnce(&B)) -> Self
Borrow<B> of a value. Read moreSource§fn tap_borrow_mut<B>(self, func: impl FnOnce(&mut B)) -> Self
fn tap_borrow_mut<B>(self, func: impl FnOnce(&mut B)) -> Self
BorrowMut<B> of a value. Read moreSource§fn tap_ref<R>(self, func: impl FnOnce(&R)) -> Self
fn tap_ref<R>(self, func: impl FnOnce(&R)) -> Self
AsRef<R> view of a value. Read moreSource§fn tap_ref_mut<R>(self, func: impl FnOnce(&mut R)) -> Self
fn tap_ref_mut<R>(self, func: impl FnOnce(&mut R)) -> Self
AsMut<R> view of a value. Read moreSource§fn tap_deref<T>(self, func: impl FnOnce(&T)) -> Self
fn tap_deref<T>(self, func: impl FnOnce(&T)) -> Self
Deref::Target of a value. Read moreSource§fn tap_deref_mut<T>(self, func: impl FnOnce(&mut T)) -> Self
fn tap_deref_mut<T>(self, func: impl FnOnce(&mut T)) -> Self
Deref::Target of a value. Read moreSource§fn tap_dbg(self, func: impl FnOnce(&Self)) -> Self
fn tap_dbg(self, func: impl FnOnce(&Self)) -> Self
.tap() only in debug builds, and is erased in release builds.Source§fn tap_mut_dbg(self, func: impl FnOnce(&mut Self)) -> Self
fn tap_mut_dbg(self, func: impl FnOnce(&mut Self)) -> Self
.tap_mut() only in debug builds, and is erased in release
builds.Source§fn tap_borrow_dbg<B>(self, func: impl FnOnce(&B)) -> Self
fn tap_borrow_dbg<B>(self, func: impl FnOnce(&B)) -> Self
.tap_borrow() only in debug builds, and is erased in release
builds.Source§fn tap_borrow_mut_dbg<B>(self, func: impl FnOnce(&mut B)) -> Self
fn tap_borrow_mut_dbg<B>(self, func: impl FnOnce(&mut B)) -> Self
.tap_borrow_mut() only in debug builds, and is erased in release
builds.Source§fn tap_ref_dbg<R>(self, func: impl FnOnce(&R)) -> Self
fn tap_ref_dbg<R>(self, func: impl FnOnce(&R)) -> Self
.tap_ref() only in debug builds, and is erased in release
builds.Source§fn tap_ref_mut_dbg<R>(self, func: impl FnOnce(&mut R)) -> Self
fn tap_ref_mut_dbg<R>(self, func: impl FnOnce(&mut R)) -> Self
.tap_ref_mut() only in debug builds, and is erased in release
builds.Source§fn tap_deref_dbg<T>(self, func: impl FnOnce(&T)) -> Self
fn tap_deref_dbg<T>(self, func: impl FnOnce(&T)) -> Self
.tap_deref() only in debug builds, and is erased in release
builds.