Crate node_crunch

Source
Expand description

NodeCrunch is a crate that allows to distribute computations among multiple nodes. The user of this crate has to implement one trait for the server and one trait for the nodes. Usually the code for the server and the node is inside the same binary and the choice if to run in server mode or node mode is done via configuration or command line argument. See some of the programs in the example folders.

Re-exports§

Modules§

  • This module contains helper structures to deal with 2D data. Array2D and Array2DChunk take care of splitting up the 2D array into chunks that can be sent to the node in order to process them.
  • This module contains the configuration for the server and the nodes. Usually code for the server and the node is shared.
  • This module contains the common error type for server and node.
  • This module contains the nc node message, trait and helper methods. To use the node you have to implement the NCNode trait that has two methods: set_initial_data() and process_data_from_server()
  • This module contains the node id and node info data structure. NodeID is just a new type pattern for a integer number. NCNodeInfo holds the node id and a time stamp for the heartbeat.
  • This module contains the nc server message, trait and helper methods To use the server you have to implement the NCServer trait that has five methods: initial_data(): This method is called once for every node when the node registers with the server. prepare_data_for_node(): This method is called when the node needs new data to process. process_data_from_node(): This method is called when the node is done with processing the data and has sent the result back to the server. heartbeat_timeout(): This method is called when the node has missed a heartbeat, usually the node is then marked as offline and the chunk of data for that node is sent to another node. finish_job(): This method is called when the job is done and all the threads are finished. Usually you want to save the results to disk in here.
  • This module contains helper functions for serializing, deserializing, sending and receiving data.