Expand description
Lamellar is an investigation of the applicability of the Rust systems programming language for HPC as an alternative to C and C++, with a focus on PGAS approaches.
§Some Nomenclature
Throughout this documentation and APIs there are a few terms we end up reusing a lot, those terms and brief descriptions are provided below:
PE
- a processing element, typically a multi threaded process, for those familiar with MPI, it corresponds to a Rank.- Commonly you will create 1 PE per psychical CPU socket on your system, but it is just as valid to have multiple PE’s per CPU
- There may be some instances where
Node
(meaning a compute node) is used instead ofPE
in these cases they are interchangeable
World
- an abstraction representing your distributed computing system- consists of N PEs all capable of communicating with one another
Team
- A subset of the PEs that exist in the worldAM
- short for Active MessageCollective Operation
- Generally means that all PEs (associated with a given distributed object) must explicitly participate in the operation, otherwise deadlock will occur.- e.g. barriers, construction of new distributed objects
One-sided Operation
- Generally means that only the calling PE is required for the operation to successfully complete.- e.g. accessing local data, waiting for local work to complete
§Features
Lamellar provides several different communication patterns and programming models to distributed applications, briefly highlighted below
§Active Messages
Lamellar allows for sending and executing user defined active messages on remote PEs in a distributed environment. User first implement runtime exported trait (LamellarAM) for their data structures and then call a procedural macro #[lamellar::am] on the implementation. The procedural macro produces all the necessary code to enable remote execution of the active message. More details can be found in the Active Messaging module documentation.
§Darcs (Distributed Arcs)
Lamellar provides a distributed extension of an Arc
called a Darc.
Darcs provide safe shared access to inner objects in a distributed environment, ensuring lifetimes and read/write accesses are enforced properly.
More details can be found in the Darc module documentation.
§PGAS abstractions
Lamellar also provides PGAS capabilities through multiple interfaces.
§LamellarArrays (Distributed Arrays)
The first is a high-level abstraction of distributed arrays, allowing for distributed iteration and data parallel processing of elements. More details can be found in the LamellarArray module documentation.
§Low-level Memory Regions
The second is a low level (unsafe) interface for constructing memory regions which are readable and writable from remote PEs. Note that unless you are very comfortable/confident in low level distributed memory (and even then) it is highly recommended you use the LamellarArrays interface More details can be found in the Memory Region module documentation.
§Network Backends
Lamellar relies on network providers called Lamellae to perform the transfer of data throughout the system. Currently three such Lamellae exist:
local
- used for single-PE (single system, single process) development (this is the default),shmem
- used for multi-PE (single system, multi-process) development, useful for emulating distributed environments (communicates through shared memory)rofi
- used for multi-PE (multi system, multi-process) distributed development, based on the Rust OpenFabrics Interface Transport Layer (ROFI) (https://github.com/pnnl/rofi).- By default support for Rofi is disabled as using it relies on both the Rofi C-library and the libfabrics library, which may not be installed on your system.
- It can be enabled by adding
features = ["enable-rofi"]
to the lamellar entry in yourCargo.toml
file
The long term goal for lamellar is that you can develop using the local
backend and then when you are ready to run distributed switch to the rofi
backend with no changes to your code.
Currently the inverse is true, if it compiles and runs using rofi
it will compile and run when using local
and shmem
with no changes.
Additional information on using each of the lamellae backends can be found below in the Running Lamellar Applications
section
§Examples
Our repository also provides numerous examples highlighting various features of the runtime: https://github.com/pnnl/lamellar-runtime/tree/master/examples
Additionally, we are compiling a set of benchmarks (some with multiple implementations) that may be helpful to look at as well: https://github.com/pnnl/lamellar-benchmarks/
Below are a few small examples highlighting some of the features of lamellar, more in-depth examples can be found in the documentation for the various features.
§Selecting a Lamellae and constructing a lamellar world instance
You can select which backend to use at runtime as shown below:
use lamellar::Backend;
fn main(){
let mut world = lamellar::LamellarWorldBuilder::new()
.with_lamellae( Default::default() ) //if "enable-rofi" feature is active default is rofi, otherwise default is `Local`
//.with_lamellae( Backend::Rofi ) //explicity set the lamellae backend to rofi,
//.with_lamellae( Backend::Local ) //explicity set the lamellae backend to local
//.with_lamellae( Backend::Shmem ) //explicity set the lamellae backend to use shared memory
.build();
}
or by setting the following envrionment variable:
LAMELLAE_BACKEND="lamellae"
where lamellae is one of local
, shmem
, or rofi
.
§Creating and executing a Registered Active Message
Please refer to the Active Messaging documentation for more details and examples
use lamellar::active_messaging::prelude::*;
#[AmData(Debug, Clone)] // `AmData` is a macro used in place of `derive`
struct HelloWorld { //the "input data" we are sending with our active message
my_pe: usize, // "pe" is processing element == a node
}
#[lamellar::am] // at a highlevel registers this LamellarAM implemenatation with the runtime for remote execution
impl LamellarAM for HelloWorld {
async fn exec(&self) {
println!(
"Hello pe {:?} of {:?}, I'm pe {:?}",
lamellar::current_pe,
lamellar::num_pes,
self.my_pe
);
}
}
fn main(){
let mut world = lamellar::LamellarWorldBuilder::new().build();
let my_pe = world.my_pe();
let num_pes = world.num_pes();
let am = HelloWorld { my_pe: my_pe };
for pe in 0..num_pes{
world.exec_am_pe(pe,am.clone()); // explicitly launch on each PE
}
world.wait_all(); // wait for all active messages to finish
world.barrier(); // synchronize with other PEs
let request = world.exec_am_all(am.clone()); //also possible to execute on every PE with a single call
world.block_on(request); //both exec_am_all and exec_am_pe return futures that can be used to wait for completion and access any returned result
}
§Creating, initializing, and iterating through a distributed array
Please refer to the LamellarArray documentation for more details and examples
use lamellar::array::prelude::*;
fn main(){
let world = lamellar::LamellarWorldBuilder::new().build();
let my_pe = world.my_pe();
let block_array = AtomicArray::<usize>::new(&world, 1000, Distribution::Block); //we also support Cyclic distribution.
block_array.dist_iter_mut().enumerate().for_each(move |(i,elem)| elem.store(i)); //simultaneosuly initialize array accross all PEs, each pe only updates its local data
block_array.wait_all();
block_array.barrier();
if my_pe == 0{
for (i,elem) in block_array.onesided_iter().into_iter().enumerate(){ //iterate through entire array on pe 0 (automatically transfering remote data)
println!("i: {} = {})",i,elem);
}
}
}
§Utilizing a Darc within an active message
Please refer to the Darc documentation for more details and examples
use lamellar::active_messaging::prelude::*;
use lamellar::darc::prelude::*;
use std::sync::atomic::{AtomicUsize,Ordering};
#[AmData(Debug, Clone)] // `AmData` is a macro used in place of `derive`
struct DarcAm { //the "input data" we are sending with our active message
cnt: Darc<AtomicUsize>, // count how many times each PE executes an active message
}
#[lamellar::am] // at a highlevel registers this LamellarAM implemenatation with the runtime for remote execution
impl LamellarAM for DarcAm {
async fn exec(&self) {
self.cnt.fetch_add(1,Ordering::SeqCst);
}
}
fn main(){
let mut world = lamellar::LamellarWorldBuilder::new().build();
let my_pe = world.my_pe();
let num_pes = world.num_pes();
let cnt = Darc::new(&world, AtomicUsize::new(0)).expect("Current PE is in world team");
for pe in 0..num_pes{
world.exec_am_pe(pe,DarcAm{cnt: cnt.clone()}); // explicitly launch on each PE
}
world.exec_am_all(DarcAm{cnt: cnt.clone()}); //also possible to execute on every PE with a single call
cnt.fetch_add(1,Ordering::SeqCst); //this is valid as well!
world.wait_all(); // wait for all active messages to finish
world.barrier(); // synchronize with other PEs
assert_eq!(cnt.load(Ordering::SeqCst),num_pes*2 + 1);
}
§Using Lamellar
Lamellar is capable of running on single node workstations as well as distributed HPC systems. For a workstation, simply copy the following to the dependency section of you Cargo.toml file:
lamellar = "0.5"
If planning to use within a distributed HPC system a few more steps may be necessary (this also works on single workstations):
- ensure Libfabric (with support for the verbs provider) is installed on your system https://github.com/ofiwg/libfabric
- set the OFI_DIR environment variable to the install location of Libfabric, this directory should contain both the following directories:
- lib
- include
- copy the following to your Cargo.toml file:
lamellar = { version = "0.5", features = ["enable-rofi"]}
For both environments, build your application as normal
cargo build (--release)
§Running Lamellar Applications
There are a number of ways to run Lamellar applications, mostly dictated by the lamellae you want to use.
§local (single-process, single system)
- directly launch the executable
cargo run --release
§shmem (multi-process, single system)
- grab the lamellar_run.sh
- Use
lamellar_run.sh
to launch your application./lamellar_run -N=2 -T=10 <appname>
N
number of PEs (processes) to launch (Default=1)T
number of threads Per PE (Default = number of cores/ number of PEs)- assumes
<appname>
executable is located at./target/release/<appname>
§rofi (multi-process, multi-system)
- allocate compute nodes on the cluster:
salloc -N 2
- launch application using cluster launcher
srun -N 2 -mpi=pmi2 ./target/release/<appname>
pmi2
library is required to grab info about the allocated nodes and helps set up initial handshakes
§Environment Variables
Lamellar exposes a number of environment variables that can used to control application execution at runtime
LAMELLAR_THREADS
- The number of worker threads used within a lamellar PEexport LAMELLAR_THREADS=10
LAMELLAE_BACKEND
- the backend used during execution. Note that if a backend is explicitly set in the world builder, this variable is ignored.- possible values
local
shmem
rofi
- possible values
LAMELLAR_MEM_SIZE
- Specify the initial size of the Runtime “RDMAable” memory pool. Defaults to 1GBexport LAMELLAR_MEM_SIZE=$((20*1024*1024*1024))
20GB memory pool- Internally, Lamellar utilizes memory pools of RDMAable memory for Runtime data structures (e.g. Darcs, OneSidedMemoryRegion,etc), aggregation buffers, and message queues. Additional memory pools are dynamically allocated across the system as needed. This can be a fairly expensive operation (as the operation is synchronous across all PEs) so the runtime will print a message at the end of execution with how many additional pools were allocated.
- if you find you are dynamically allocating new memory pools, try setting
LAMELLAR_MEM_SIZE
to a larger value
- if you find you are dynamically allocating new memory pools, try setting
- Note: when running multiple PEs on a single system, the total allocated memory for the pools would be equal to
LAMELLAR_MEM_SIZE * number of processes
Re-exports§
pub extern crate serde_with;
pub use lamellar_env::LamellarEnv;
Modules§
- Active Messages are a computing model where messages contain both data (that you want to compute something with) and metadata that tells the message how to process its data when it arrives at its destination, e.g. a function pointer. The Wikipedia Page https://en.wikipedia.org/wiki/Active_message provides a short overview.
- LamellarArrays provide a safe and high-level abstraction of a distributed array.
- A trait for accessing various data about the current lamellar envrionment
- Memory regions are unsafe low-level abstractions around shared memory segments that have been allocated by a lamellae provider.
Structs§
- A group of active messages that can be executed in parallel the active messages do not need the be the same type, but they must all return the unit type i.e.
()
Future implementations will relax this restriction, so that they only need to return the same type. - A grouping of PE’s forming a team using a “block” based distribution pattern.
- An error that occurs when trying to access a PE that does not exist on a team/subteam
- An abstraction for representing a set of active messages as single group.
- An abstraction used to group PEs into distributed computational units.
- An abstraction representing all the PE’s (processing elements) within a given distributed execution.
- An implementation of the Builder design pattern, used to construct an instance of a LamellarWorld.
- A grouping of PE’s forming a team using a “strided” based distribution pattern.
- A struct to hold the requests for a TypedAmGroup request corresponding to a single PE
- A struct to hold the results of a TypedAmGroup request corresponding to a single PE
Enums§
- Contains the Result of an AM Group Request
- The list of available lamellae backends, used to specify how data is transfered between PEs
- This enum is used to specify the type of AmGroup request
- The available worker thread scheduling algorithms
- Hold the results from a Typed AM group request
Traits§
- An abstraction which represents the PEs that are associated with a Lamellar team