Expand description
Control operations (engine/io_struct.py): cache flush, pause, memory
release, weight updates, profiling and the like.
A frontend sends one to every engine as a CONTROL message; each engine
answers with the matching reply on its output socket, beside the output
batches. Replies carry no request id: an engine answers each kind of
operation in the order it received them. Every struct is tagged and
starts with the two BaseReq fields.
Structs§
- Destroy
Weights Update Group ReqInput - Leave a weight-update group.
- Destroy
Weights Update Group ReqOutput DestroyWeightsUpdateGroupReqInput’s reply.- Flush
Cache ReqInput - Drop the prefix caches.
- Flush
Cache ReqOutput FlushCacheReqInput’s reply.- GetInternal
State Req - The engine’s internal state.
- GetInternal
State ReqOutput GetInternalStateReq’s reply.- Init
Weights Update Group ReqInput - Join the trainer’s weight-update group.
- Init
Weights Update Group ReqOutput InitWeightsUpdateGroupReqInput’s reply.- IsScheduler
Paused ReqInput - Whether scheduling is paused.
- IsScheduler
Paused ReqOutput IsSchedulerPausedReqInput’s reply.- IsSleeping
ReqInput - Whether the engine released its memory.
- IsSleeping
ReqOutput IsSleepingReqInput’s reply.- Pause
Scheduler ReqInput - Stop scheduling:
abortaborts what runs,waitlets it finish,keepfreezes it in place. - Pause
Scheduler ReqOutput PauseSchedulerReqInput’s reply, once the pause took hold.- Profile
Req - Start (
kind1) or stop (kind2) the profiler. - Profile
ReqOutput ProfileReq’s reply.- Rebalance
Experts ReqInput - Start one online expert rebalance now.
- Rebalance
Experts ReqOutput RebalanceExpertsReqInput’s reply.- Release
Memory Occupation ReqInput - Release GPU memory (
weights,kv_cache; all whenNone). - Release
Memory Occupation ReqOutput ReleaseMemoryOccupationReqInput’s reply.- Resume
Memory Occupation ReqInput - Take released GPU memory back (all released regions when
None). - Resume
Memory Occupation ReqOutput ResumeMemoryOccupationReqInput’s reply.- Resume
Scheduler ReqInput - Resume scheduling after a pause.
- Resume
Scheduler ReqOutput ResumeSchedulerReqInput’s reply.- SetInternal
State Req - Change server arguments at run time.
- SetInternal
State ReqOutput SetInternalStateReq’s reply.- Update
Weight From Disk ReqInput - Load new weights from a checkpoint on disk.
- Update
Weight From Disk ReqOutput UpdateWeightFromDiskReqInput’s reply.- Update
Weights From Distributed ReqInput - Receive new weights from the trainer’s NCCL group.
- Update
Weights From Distributed ReqOutput UpdateWeightsFromDistributedReqInput’s reply.- Update
Weights From Mooncake ReqInput - Load one committed version from a Mooncake weight store.
- Update
Weights From Mooncake ReqOutput UpdateWeightsFromMooncakeReqInput’s reply.- Update
Weights From Tensor ReqInput - Load weights from serialized tensors; the engines refuse it.
- Update
Weights From Tensor ReqOutput UpdateWeightsFromTensorReqInput’s reply.
Enums§
- Control
Output - A control operation’s reply.
- Control
Request - A control operation, the payload of a CONTROL message.
Constants§
- PROFILE_
START ProfileReq.kindthat starts the profiler.- PROFILE_
STOP ProfileReq.kindthat stops it.