Struct actix_raft::Raft

pub struct Raft<D: AppData, R: AppDataResponse, E: AppError, N: RaftNetwork<D>, S: RaftStorage<D, R, E>> { /* fields omitted */ }

An actor which implements the Raft protocol's core business logic.

For more information on the Raft protocol, see the specification here: https://raft.github.io/raft.pdf (pdf warning).

The beginning of §5, the spec has a condensed summary of the Raft consensus algorithm. This crate, and especially this actor, attempts to follow the terminology and nomenclature used there as precisely as possible to aid in understanding this system.

api

This actor's API is broken up into 3 different layers, all based on message handling. In order to effectively use this actor, only these 3 layers need to be considered.

network

The network interface of the parent application is responsible for providing a conduit to exchange the various messages types defined in this system. The RaftNetwork trait defines the interface needed for being able to allow Raft cluster members to be able to communicate with each other. In addition to the RaftNetwork trait, applications are expected to provide and interface for their clients to be able to submit data which needs to be managed by Raft.

raft rpc messages

These are Raft request PRCs coming from other nodes of the cluster. They are defined in the messages module of this crate. They are AppendEntriesRequest, VoteRequest & InstallSnapshotRequest. This actor will use the RaftNetwork impl of the parent application to send RPCs to other nodes.

The application's networking layer must decode these message types and pass them over to the appropriate handler on this type, await the response, and then send the response back over the wire to the caller.

client request messages

These are messages coming from an application's clients, represented by the messages::ClientPayload type. When the message type's handler is called a future will be returned which will resolve with the appropriate response type. Only data mutating messages should ever need to go through Raft. The contentsof these messages are entirely specific to your application.

storage

The storage interface is typically going to be the most involved as this is where your application really exists. SQL, NoSQL, mutable, immutable, KV, append only ... whatever your application's data model, this is where it comes to life.

The storage interface is defined by the RaftStorage trait. Depending on the data storage system being used, the actor my be sync or async. It just needs to implement handlers for the needed actix message types.

admin

These are admin commands which may be issued to a Raft node in order to influence it in ways outside of the normal Raft lifecycle. Dynamic membership changes and cluster initialization are the main commands of this layer.

Methods

impl<D: AppData, R: AppDataResponse, E: AppError, N: RaftNetwork<D>, S: RaftStorage<D, R, E>> Raft<D, R, E, N, S>

pub fn new(
    id: NodeId,
    config: Config,
    network: Addr<N>,
    storage: Addr<S::Actor>,
    metrics: Recipient<RaftMetrics>
) -> Self

Create a new Raft instance.

This actor will need to be started after instantiation, which must be done within a running actix system.

Trait Implementations

impl<D: AppData, R: AppDataResponse, E: AppError, N: RaftNetwork<D>, S: RaftStorage<D, R, E>> Actor for Raft<D, R, E, N, S>

type Context = Context<Self>

Actor execution context type

fn started(&mut self, ctx: &mut Self::Context)

The initialization routine for this actor.

fn stopping(&mut self, ctx: &mut Self::Context) -> Running

Called after an actor is in Actor::Stopping state.

fn stopped(&mut self, ctx: &mut Self::Context)

Called after an actor is stopped.

fn start(self) -> Addr<Self> where
    Self: Actor<Context = Context<Self>>, 

Start a new asynchronous actor, returning its address.

fn start_default() -> Addr<Self> where
    Self: Actor<Context = Context<Self>> + Default, 

Construct and start a new asynchronous actor, returning its address.

fn start_in_arbiter<F>(arb: &Arbiter, f: F) -> Addr<Self> where
    F: FnOnce(&mut Context<Self>) -> Self + Send + 'static,
    Self: Actor<Context = Context<Self>>, 

Start new actor in arbiter's thread.

fn create<F>(f: F) -> Addr<Self> where
    F: FnOnce(&mut Context<Self>) -> Self + 'static,
    Self: Actor<Context = Context<Self>>, 

Start a new asynchronous actor given a Context.

impl<D: AppData, R: AppDataResponse, E: AppError, N: RaftNetwork<D>, S: RaftStorage<D, R, E>> Handler<InitWithConfig> for Raft<D, R, E, N, S>

type Result = ResponseActFuture<Self, (), InitWithConfigError>

The type of value that this handler will return.

fn handle(
    &mut self,
    msg: InitWithConfig,
    ctx: &mut Self::Context
) -> Self::Result

An admin message handler invoked exclusively for cluster formation.

This command will work for single-node or multi-node cluster formation. This command should be called with all discovered nodes which need to be part of cluster.

This command will be rejected if the node is not at index 0 & in the NonVoter state, as either of those constraints being false indicates that the cluster is already formed and in motion.

This routine will set the given config as the active config, only in memory, and will start an election.

All nodes must issue this command at startup, as they will not be able to vote for other nodes until they appear in their config. This handler will ensure that it is safe to execute this command.

Once a node becomes leader and detects that its index is 0, it will commit a new config entry (instead of the normal blank entry created by new leaders).

If a race condition takes place where two nodes persist an initial config and start an election, whichever node becomes leader will end up committing its entries to the cluster.

impl<D: AppData, R: AppDataResponse, E: AppError, N: RaftNetwork<D>, S: RaftStorage<D, R, E>> Handler<ProposeConfigChange<D, R, E>> for Raft<D, R, E, N, S>

type Result = ResponseActFuture<Self, (), ProposeConfigChangeError<D, R, E>>

The type of value that this handler will return.

fn handle(
    &mut self,
    msg: ProposeConfigChange<D, R, E>,
    ctx: &mut Self::Context
) -> Self::Result

An admin message handler invoked to trigger dynamic cluster configuration changes. See §6.

impl<D: AppData, R: AppDataResponse, E: AppError, N: RaftNetwork<D>, S: RaftStorage<D, R, E>> Handler<AppendEntriesRequest<D>> for Raft<D, R, E, N, S>

type Result = ResponseActFuture<Self, AppendEntriesResponse, ()>

The type of value that this handler will return.

fn handle(
    &mut self,
    msg: AppendEntriesRequest<D>,
    ctx: &mut Self::Context
) -> Self::Result

An RPC invoked by the leader to replicate log entries (§5.3); also used as heartbeat (§5.2).

This method implements the append entries algorithm and upholds all of the safety checks detailed in §5.3.

Implementation overview from spec:

1. Reply false if term is less than node's current term (§5.1).
2. Reply false if log doesn’t contain an entry at prev_log_index whose term matches prev_log_term (§5.3).
3. If an existing entry conflicts with a new one (same index but different terms), delete the existing entry and all that follow it (§5.3).
4. Append any new entries not already in the log.
5. If leader_commit is greater than node's commit index, set nodes commit index to min(leader_commit, index of last new entry).

The essential goal of this algorithm is that the receiver (the node on which this method is being executed) must find the exact entry in its log specified by the RPC's last index and last term fields, and then begin writing the new entries thereafter.

When the receiver can not find the entry specified in the RPC's prev index & prev term fields, it will respond with a failure to the leader. This implementation of Raft includes the conflicting term optimization which is intended to reduce the number of rejected append entries RPCs from followers which are lagging behind, which is detailed in §5.3. In such cases, if the Raft cluster is configured with a snapshot policy other than Disabled, the leader will make a determination if an InstallSnapshot RPC should be sent to this node.

In Raft, the leader handles inconsistencies by forcing the followers’ logs to duplicate its own. This means that conflicting entries in follower logs will be overwritten with entries from the leader’s log. §5.4 details the safety of this protocol. It is important to note that logs which are committed will not be overwritten. This is a critical feature of Raft.

Raft also gurantees that only logs which have been comitted may be applied to the state machine, which ensures that there will never be a case where a log needs to be reverted after being applied to the state machine.

inconsistency example

Followers may receive valid append entries requests from leaders, append them, respond, and before the leader is able to replicate the entries to a majority of nodes, the leader may die, a new leader may be elected which does not have the same entries, as they were not replicated to a majority of followers, and the new leader will proceeed to overwrite the inconsistent entries.

impl<D: AppData, R: AppDataResponse, E: AppError, N: RaftNetwork<D>, S: RaftStorage<D, R, E>> Handler<ClientPayload<D, R, E>> for Raft<D, R, E, N, S>

type Result = ResponseActFuture<Self, ClientPayloadResponse<R>, ClientError<D, R, E>>

The type of value that this handler will return.

fn handle(
    &mut self,
    msg: ClientPayload<D, R, E>,
    _: &mut Self::Context
) -> Self::Result

Handle client requests.

impl<D: AppData, R: AppDataResponse, E: AppError, N: RaftNetwork<D>, S: RaftStorage<D, R, E>> Handler<InstallSnapshotRequest> for Raft<D, R, E, N, S>

type Result = ResponseActFuture<Self, InstallSnapshotResponse, ()>

The type of value that this handler will return.

fn handle(
    &mut self,
    msg: InstallSnapshotRequest,
    ctx: &mut Self::Context
) -> Self::Result

Invoked by leader to send chunks of a snapshot to a follower (§7).

Leaders always send chunks in order. It is important to note that, according to the Raft spec, a log may only have one snapshot at any time. As snapshot contents are application specific, the Raft log will only store a pointer to the snapshot file along with the index & term.

See the storage::InstallSnapshot type for implementaion details.

impl<D: AppData, R: AppDataResponse, E: AppError, N: RaftNetwork<D>, S: RaftStorage<D, R, E>> Handler<VoteRequest> for Raft<D, R, E, N, S>

type Result = ResponseActFuture<Self, VoteResponse, ()>

The type of value that this handler will return.

fn handle(&mut self, msg: VoteRequest, ctx: &mut Self::Context) -> Self::Result

An RPC invoked by candidates to gather votes (§5.2).

Receiver implementation:

1. Reply false if term is less than receiver's current term (§5.1).
2. If receiver has not cast a vote for the current term or it voted for candidate_id, and candidate’s log is atleast as up-to-date as receiver’s log, grant vote (§5.2, §5.4).