Expand description
Portable model capabilities, runtime-state accounting, and admission policy. Portable model capabilities, runtime-state accounting, and admission policy.
Structs§
- Admission
- Detailed successful admission.
- Admission
Request - One pre-generation admission request.
- Available
Memory - System memory usable as an admission signal.
- Input
Modalities - Model inputs accepted by a prepared model.
- Input
Token Count - Accounting for a tokenized or backend-prepared input.
- Model
Capabilities - Capabilities derived from validated model configuration.
- Runtime
State Estimate - Persistent and transient runtime-state estimate for one request.
- Sliding
Window Layer Count - Sliding-attention layer count sharing one retained window.
- State
Memory Assumptions - Floating-dtype and request assumptions used by state estimation.
- State
Memory Layout - Memory-accounting view of an executable runtime-state layout.
- Static
Memory Report - Static checkpoint and current residency observations.
Enums§
- Admission
Rejection - Structured reason a request was rejected before generation.
- Admission
Result - Admission outcome.
- Cache
State Strategy - Persistent decoder-state strategy used by a model.
- Capability
Error - Structured capability and accounting failures.
- Estimation
Completeness - Coverage of a runtime-state estimate.
- Physical
Memory Semantics - Physical relationship between logical host and device tiers.
Functions§
- apply_
admission_ policy - Applies context and memory policy to an already-computed state estimate.
- estimate_
runtime_ state - Estimates request state from exact executable layer policies.