pub struct ParameterServer { /* private fields */ }Expand description
Centralized parameter server: rank 0 collects, averages, and broadcasts gradients (or parameters).
Use this instead of AllReduceAggregator when you want a star topology
(all workers communicate only with rank 0). This is simpler to reason about
and works well when the coordinator has high bandwidth, but can become a
bottleneck at large scale compared to ring all-reduce.
broadcast_params– rank 0 sends the current parameters to all workers (useful at initialisation or after a checkpoint restore).reduce_gradients– workers send local gradients to rank 0; rank 0 averages them and broadcasts the result back.
Implementations§
Source§impl ParameterServer
impl ParameterServer
pub fn new(config: DistributedConfig, transport: Box<dyn Transport>) -> Self
Sourcepub fn broadcast_params(
&self,
params: &[Tensor],
) -> Result<Vec<Tensor>, ModelError>
pub fn broadcast_params( &self, params: &[Tensor], ) -> Result<Vec<Tensor>, ModelError>
Rank 0 broadcasts params to all workers; workers receive and return them.
Sourcepub fn reduce_gradients(
&self,
grads: &[Tensor],
) -> Result<Vec<Tensor>, ModelError>
pub fn reduce_gradients( &self, grads: &[Tensor], ) -> Result<Vec<Tensor>, ModelError>
Workers send gradients to rank 0; rank 0 averages and returns result.
Auto Trait Implementations§
impl Freeze for ParameterServer
impl !RefUnwindSafe for ParameterServer
impl Send for ParameterServer
impl !Sync for ParameterServer
impl Unpin for ParameterServer
impl UnsafeUnpin for ParameterServer
impl !UnwindSafe for ParameterServer
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more