Crate node_crunch

NodeCrunch is a crate that allows to distribute computations among multiple nodes. The user of this crate has to implement one trait for the server and one trait for the nodes. Usually the code for the server and the node is inside the same binary and the choice if to run in server mode or node mode is done via configuration or command line argument. See some of the programs in the example folders.

Re-exports

pub use nc_server::NCServer;

pub use nc_server::NCJobStatus;

pub use nc_server::NCServerStarter;

pub use nc_node::NCNode;

pub use nc_node::NCNodeStarter;

pub use nc_node_info::NodeID;

pub use nc_error::NCError;

pub use nc_config::NCConfiguration;

pub use nc_util::nc_decode_data;

pub use nc_util::nc_decode_data2;

pub use nc_util::nc_encode_data;

pub use array2d::Array2D;

pub use array2d::Array2DChunk;

pub use array2d::ChunkList;

pub use array2d::Chunk;

Modules

array2d

This module contains helper structures to deal with 2D data. Array2D and Array2DChunk take care of splitting up the 2D array into chunks that can be sent to the node in order to process them.

nc_config

This module contains the configuration for the server and the nodes. Usually code for the server and the node is shared.

nc_error

This module contains the common error type for server and node.

nc_node

This module contains the nc node message, trait and helper methods. To use the node you have to implement the NCNode trait that has two methods: set_initial_data() and process_data_from_server()

nc_node_info

This module contains the node id and node info data structure. NodeID is just a new type pattern for a integer number. NCNodeInfo holds the node id and a time stamp for the heartbeat.

nc_server

This module contains the nc server message, trait and helper methods To use the server you have to implement the NCServer trait that has five methods: initial_data(): This method is called once for every node when the node registers with the server. prepare_data_for_node(): This method is called when the node needs new data to process. process_data_from_node(): This method is called when the node is done with processing the data and has sent the result back to the server. heartbeat_timeout(): This method is called when the node has missed a heartbeat, usually the node is then marked as offline and the chunk of data for that node is sent to another node. finish_job(): This method is called when the job is done and all the threads are finished. Usually you want to save the results to disk in here.

nc_util

This module contains helper functions for serializing, deserializing, sending and receiving data.