Crate lamellar

Expand description

Lamellar is an investigation of the applicability of the Rust systems programming language for HPC as an alternative to C and C++, with a focus on PGAS approaches.

§Some Nomenclature

Throughout this documentation and APIs there are a few terms we end up reusing a lot, those terms and brief descriptions are provided below:

PE - a processing element, typically a multi threaded process, for those familiar with MPI, it corresponds to a Rank.
- Commonly you will create 1 PE per physical CPU socket on your system, but it is just as valid to have multiple PE’s per CPU
- There may be some instances where Node (meaning a compute node) is used instead of PE in these cases they are interchangeable
World - an abstraction representing your distributed computing system
- consists of N PEs all capable of communicating with one another
Team - A subset of the PEs that exist in the world
AM - short for Active Message
Collective Operation - Generally means that all PEs (associated with a given distributed object) must explicitly participate in the operation, otherwise deadlock will occur.
- e.g. barriers, construction of new distributed objects
One-sided Operation - Generally means that only the calling PE is required for the operation to successfully complete.
- e.g. accessing local data, waiting for local work to complete

§Features

Lamellar provides several different communication patterns and programming models to distributed applications, briefly highlighted below

§Active Messages

Lamellar allows for sending and executing user defined active messages on remote PEs in a distributed environment. User first implement runtime exported trait (LamellarAM) for their data structures and then call a procedural macro #[lamellar::am] on the implementation. The procedural macro produces all the necessary code to enable remote execution of the active message. More details can be found in the Active Messaging module documentation.

§Darcs (Distributed Arcs)

Lamellar provides a distributed extension of an Arc called a Darc. Darcs provide safe shared access to inner objects in a distributed environment, ensuring lifetimes and read/write accesses are enforced properly. More details can be found in the Darc module documentation.

§PGAS abstractions

Lamellar also provides PGAS capabilities through multiple interfaces.

§LamellarArrays (Distributed Arrays)

The first is a high-level abstraction of distributed arrays, allowing for distributed iteration and data parallel processing of elements. More details can be found in the LamellarArray module documentation.

§Low-level Memory Regions

The second is a low level (unsafe) interface for constructing memory regions which are readable and writable from remote PEs. Note that unless you are very comfortable/confident in low level distributed memory (and even then) it is highly recommended you use the LamellarArrays interface More details can be found in the Memory Region module documentation.

§Network Backends

Lamellar relies on network providers called Lamellae to perform the transfer of data throughout the system. Currently several such Lamellae exist (some require feature flags to enable):

local - used for single-PE (single system, single process) development (this is the default),
shmem - used for multi-PE (single system, multi-process) development, useful for emulating distributed environments (communicates through shared memory)
rofi-c - used for multi-PE (multi system, multi-process) distributed development, based on the Rust OpenFabrics Interface Transport Layer (ROFI) (https://github.com/pnnl/rofi).
- Requires the Rofi C-library and libfabrics; enable with features = ["enable-rofi-c"] in your Cargo.toml
libfabric - multi-PE distributed backend using libfabrics directly; enable with features = ["enable-libfabric"]
libfabric-async - async variant of the libfabric backend; enable with features = ["enable-libfabric-async"]
ucx - multi-PE distributed backend using UCX; enable with features = ["enable-ucx"]

The long term goal for lamellar is that you can develop using the local backend and then when you are ready to run distributed switch to the rofi backend with no changes to your code. Currently the inverse is true, if it compiles and runs using rofi it will compile and run when using local and shmem with no changes.

Additional information on using each of the lamellae backends can be found below in the Running Lamellar Applications section

§Environment Variables

Lamellar has a number of environment variables that can be used to configure the runtime. please see the Environment Variables module documentation for more details

§Examples

Our repository also provides numerous examples highlighting various features of the runtime: https://github.com/pnnl/lamellar-runtime/tree/master/examples

Additionally, we are compiling a set of benchmarks (some with multiple implementations) that may be helpful to look at as well: https://github.com/pnnl/lamellar-benchmarks/

Below are a few small examples highlighting some of the features of lamellar, more in-depth examples can be found in the documentation for the various features.

§Selecting a Lamellae and constructing a lamellar world instance

You can select which backend to use at runtime as shown below:

use lamellar::Backend;
fn main(){
 let mut world = lamellar::LamellarWorldBuilder::new()
        .with_lamellae( Default::default() ) //if "enable-rofi-c" feature is active default is rofi, otherwise  default is `Local`
        //.with_lamellae( Backend::RofiC ) //explicity set the lamellae backend to rofi,
        //.with_lamellae( Backend::Local ) //explicity set the lamellae backend to local
        //.with_lamellae( Backend::Shmem ) //explicity set the lamellae backend to use shared memory
        .build();
}

or by setting the following environment variable: LAMELLAR_BACKEND="lamellae" where lamellae is one of local, shmem, or rofi.

§Creating and executing a Registered Active Message

Please refer to the Active Messaging documentation for more details and examples

use lamellar::active_messaging::prelude::*;

#[AmData(Debug, Clone)] // `AmData` is a macro used in place of `derive`
struct HelloWorld { //the "input data" we are sending with our active message
    my_pe: usize, // "pe" is processing element == a node
}

#[lamellar::am] // at a highlevel registers this LamellarAM implementation with the runtime for remote execution
impl LamellarAM for HelloWorld {
    async fn exec(&self) {
        println!(
            "Hello pe {:?} of {:?}, I'm pe {:?}",
            lamellar::current_pe,
            lamellar::num_pes,
            self.my_pe
        );
    }
}

fn main(){
    let mut world = lamellar::LamellarWorldBuilder::new().build();
    let my_pe = world.my_pe();
    let num_pes = world.num_pes();
    let am = HelloWorld { my_pe: my_pe };
    for pe in 0..num_pes{
        let _ = world.spawn_am_pe(pe,am.clone()); // explicitly launch on each PE
    }
    world.wait_all(); // wait for all active messages to finish
    world.barrier();  // synchronize with other PEs
    let request = world.exec_am_all(am.clone()); //also possible to execute on every PE with a single call
    request.block(); //exec_am_all returns a future that can be used to wait for completion and access any returned result
}

§Creating, initializing, and iterating through a distributed array

Please refer to the LamellarArray documentation for more details and examples

use lamellar::array::prelude::*;

fn main(){
    let world = lamellar::LamellarWorldBuilder::new().build();
    let my_pe = world.my_pe();
    let block_array = AtomicArray::<usize>::new(&world, 1000, Distribution::Block).block(); //we also support Cyclic distribution.
    let _ =block_array.dist_iter_mut().enumerate().for_each(move |(i,elem)| elem.store(i)).block(); //simultaneously initialize array across all PEs, each pe only updates its local data
    block_array.barrier();
    if my_pe == 0{
        for (i,elem) in block_array.onesided_iter().into_iter().enumerate(){ //iterate through entire array on pe 0 (automatically transferring remote data)
            println!("i: {} = {})",i,elem);
        }
    }
}

§Utilizing a Darc within an active message

Please refer to the Darc documentation for more details and examples

 use lamellar::active_messaging::prelude::*;
 use lamellar::darc::prelude::*;
 use std::sync::atomic::{AtomicUsize,Ordering};

 #[AmData(Debug, Clone)] // `AmData` is a macro used in place of `derive`
 struct DarcAm { //the "input data" we are sending with our active message
     cnt: Darc<AtomicUsize>, // count how many times each PE executes an active message
 }

 #[lamellar::am] // at a highlevel registers this LamellarAM implementation with the runtime for remote execution
 impl LamellarAM for DarcAm {
     async fn exec(&self) {
         self.cnt.fetch_add(1,Ordering::SeqCst);
     }
 }

 fn main(){
     let mut world = lamellar::LamellarWorldBuilder::new().build();
     let my_pe = world.my_pe();
     let num_pes = world.num_pes();
     let cnt = Darc::new(&world, AtomicUsize::new(0)).block().expect("Current PE is in world team");
     for pe in 0..num_pes{
         let _ = world.spawn_am_pe(pe,DarcAm{cnt: cnt.clone()}); // explicitly launch on each PE
     }
     let _ = world.spawn_am_all(DarcAm{cnt: cnt.clone()}); //also possible to execute on every PE with a single call
     cnt.fetch_add(1,Ordering::SeqCst); //this is valid as well!
     world.wait_all(); // wait for all active messages to finish
     world.barrier();  // synchronize with other PEs
     assert_eq!(cnt.load(Ordering::SeqCst),num_pes*2 + 1);
 }

§Using Lamellar

Lamellar is capable of running on single node workstations as well as distributed HPC systems. For a workstation, simply copy the following to the dependency section of you Cargo.toml file:

lamellar = "0.8.0"

If planning to use within a distributed HPC system copy the following to your Cargo.toml file:

lamellar = { version = "0.8.0", features = ["enable-rofi-c"]}

NOTE: as of Lamellar 0.6.1 It is no longer necessary to manually install Libfabric, the build process will now try to automatically build libfabric for you. If this process fails, it is still possible to pass in a manual libfabric installation via the OFI_DIR environment variable.

For both environments, build your application as normal

cargo build (--release)

§Running Lamellar Applications

A Lamellar application’s main function must be annotated with #[lamellar::main] (see examples above), which is responsible for launching the requested number of PEs. You still construct the world yourself (e.g. via LamellarWorldBuilder) inside main.

The generated main detects whether it’s already running as a launched PE (e.g. under PRRTE/srun); if not, it re-executes itself under a launcher (prterun by default, or srun with the use-srun feature). A single cargo run/binary invocation transparently becomes a multi-PE job – no separate lamellar_run.sh step is needed.

§local / shmem (single system, one or many processes)

directly launch the executable
- cargo run --release
to run multiple PEs on a single system (shared-memory backend), add launcher args after a ---separated section:
- LAMELLAR_BACKEND=shmem cargo run --release -- <app args> -- --pes 4

§distributed (multi-process, multi-system)

allocate compute nodes on the cluster:
- salloc -N 2
run your application the same way as above – #[lamellar::main] handles invoking prterun/srun across the allocated nodes; select the distributed backend via LAMELLAR_BACKEND (e.g. rofi_c, libfabric, ucx) or the corresponding Backend::* variant in the world builder:
- cargo run --release -- <app args> -- --pes 8 --pes-per-node 4

See the README for the full set of launch flags (--nodes, --lamellae, --cmd-queue, etc.).

Re-exports§

pub extern crate serde;
pub extern crate serde_with;

Re-exports§

pub use lamellar_env::LamellarEnv;
pub use env_var::available_backends;
pub use env_var::compiled_default_backend;
pub use parking_lot;
pub use ::tracing;
pub use ::tracing_subscriber;
pub use async_trait;
pub use futures_util;
pub use lamellar_prof;
pub use inventory;
pub use bincode;
pub use custom_derive;
pub use newtype_derive;
pub use async_std;
pub use active_messaging::prelude::*;
pub use array::prelude::*;
pub use darc::prelude::*;
pub use memregion::prelude::*;

Modules§

active_messaging: Active Messages are a computing model where messages contain both data (that you want to compute something with) and metadata that tells the message how to process its data when it arrives at its destination, e.g. a function pointer. The Wikipedia Page https://en.wikipedia.org/wiki/Active_message provides a short overview.
array: LamellarArrays provide a safe and high-level abstraction of a distributed array.
callsite: Callsites represent the source locations from which spans or events originate.
darc: Distributed Atomic Reference Counter– a distributed extension of an Arc called a Darc. The atomic reference counter, Arc, is a backbone of safe concurrent programming in Rust, and, in particular, shared ownership.
de: Generic data structure deserialization framework.
dispatcher: Dispatches trace events to Subscribers.
env_var: Lamellar uses a number of environment variables to configure its behavior the following variables are supported along with a brief description and default value
event: Events represent single points in time during the execution of a program.
field: Span and Event key-value data.
instrument: Attach a span to a std::future::Future.
lamellar_env: A trait for accessing various data about the current lamellar environment
level_filters: Trace verbosity level filtering.
memregion: Memory regions are unsafe low-level abstractions around shared memory segments that have been allocated by a lamellae provider.
metadata: Metadata describing trace data.
ser: Generic data structure serialization framework.
span: Spans represent periods of time in which a program was executing in a particular context.
subscriber: Collects and records trace data.

Macros§

debug: Constructs an event at the debug level.
debug_span: Constructs a span at the debug level.
enabled: Checks whether a span or event is enabled based on the provided metadata.
error: Constructs an event at the error level.
error_span: Constructs a span at the error level.
event: Constructs a new Event.
event_enabled: Tests whether an event with the specified level and target would be enabled.
fini_prof: Print the accumulated profiling data for the current thread and reset state. Call at the end of the region you want to profile (e.g. end of main). Only active when enable-prof is enabled.
forward_to_deserialize_any: Helper macro when implementing the Deserializer part of a new data format for Serde.
info: Constructs an event at the info level.
info_span: Constructs a span at the info level.
init_prof: Emit RAII-stack profiling infrastructure into the calling crate.
init_prof_bt: Emit backtrace-based profiling infrastructure into the calling crate.
metadata: Statically constructs new span metadata.
record_all: Records multiple values on a span in a single call. As with recording individual values, all fields must be declared when the span is created.
span: Constructs a new span.
span_enabled: Tests whether a span with the specified level and target would be enabled.
trace: Constructs an event at the trace level.
trace_span: Constructs a span at the trace level.
warn: Constructs an event at the warn level.
warn_span: Constructs a span at the warn level.

Structs§

AmGroup: A group of active messages that can be executed in parallel the active messages do not need to be the same type, but they must all return the unit type i.e. () Future implementations will relax this restriction, so that they only need to return the same type.
AtomicCompareExchangeOpHandle: A task handle for a remote atomic compare-and-exchange operation.
AtomicFetchOpHandle: A task handle for a remote atomic fetch-and-modify operation (e.g. fetch_add, fetch_and, etc.) that returns the previous value of the modified element.
AtomicOpHandle: A task handle for raw RMDA (put/get) operation
BlockedArch: A grouping of PE’s forming a team using a “block” based distribution pattern.
Dispatch: Dispatch trace data to a Subscriber.
Event: Events represent single points in time where something occurred during the execution of a program.
Id: Identifies a span within the context of a subscriber.
IdError: An error that occurs when trying to access a PE that does not exist on a team/subteam
LamellarTask: A LamellarTask is a wrapper around a future that is being executed by the Lamellar scheduler. LamellarTasks can be either awaited or blocked on.
LamellarTaskGroup: An abstraction for representing a set of active messages as single group.
LamellarTeam: An abstraction used to group PEs into distributed computational units.
LamellarWorld: An abstraction representing all the PE’s (processing elements) within a given distributed execution.
LamellarWorldBuilder: An implementation of the Builder design pattern, used to construct an instance of a LamellarWorld.
Level: Describes the level of verbosity of a span or event.
Metadata: Metadata describing a span or event.
RdmaHandle: A task handle for raw RMDA (put/get) operation
Span: A handle representing a span, with the capability to enter the span if it exists.
StridedArch: A grouping of PE’s forming a team using a “strided” based distribution pattern.
TypedAmGroupBatchReq: A struct to hold the requests for a TypedAmGroup request corresponding to a single PE
TypedAmGroupBatchResult: A struct to hold the results of a TypedAmGroup request corresponding to a single PE

Enums§

AmGroupResult: Contains the Result of an AM Group Request
Backend: The list of available lamellae backends, used to specify how data is transferred between PEs
BaseAmGroupReq: This enum is used to specify the type of AmGroup request
BroadcastInput: Input type for broadcast collective operations, indicating whether this PE is the root (source) or not.
ExecutorType: Indicates the executor backend Default is a work stealing executor If the “tokio-executor” feature is enabled,the tokio executor can also be used allowing seamless integration with tokio based applications
RootOrLamellarBuffer: Input type for gather-to-root collective operations, providing the root PE’s destination buffer or a non-root PE’s root index.
RootSrcOrLamellarBuffer: Input type for scatter/broadcast operations where the root PE provides a source index and non-root PEs provide a destination buffer.
ScatterInput: Input type for scatter collective operations, indicating whether this PE is the root (source) or not.
TypedAmGroupResult: Hold the results from a Typed AM group request

Constants§

LAMELLAR_THREAD_ID: A thread-local unique identifier for the current Lamellar worker thread.

Traits§

Callsite: Trait implemented by callsites.
Deserialize: A data structure that can be deserialized from any data format supported by Serde.
Deserializer: A data format that can deserialize any data structure supported by Serde.
Instrument: Attaches spans to a std::future::Future.
LamellarArch: An abstraction which represents the PEs that are associated with a Lamellar team
Serialize: A data structure that can be serialized into any data format supported by Serde.
Serializer: A data format that can serialize any data structure supported by Serde.
Subscriber: Trait representing the functions required to collect trace data.
Value: A field value of an erased type.

Functions§

deserialize: Wrapper function for deserializing data.
serialize: Serializes an object into a byte vector.
serialize_into: Wrapper function for serializing an object into a buffer.
serialized_size: Returns the size of serialized data without allocating.

Attribute Macros§

instrumentattributes: Instruments a function to create and enter a tracing span every time the function is called.
prof: Instrument a function or all methods in an impl block for profiling.
prof_all: Instrument all methods in an impl block for profiling.

Derive Macros§

Deserializeserde_derive: Derive macro available if serde is built with features = ["derive"].
Dist
Serializeserde_derive: Derive macro available if serde is built with features = ["derive"].

Crate lamellar

Crate lamellar Copy item path

§Some Nomenclature

§Features

§Active Messages

§Darcs (Distributed Arcs)

§PGAS abstractions

§LamellarArrays (Distributed Arrays)

§Low-level Memory Regions

§Network Backends

§Environment Variables

§Examples

§Selecting a Lamellae and constructing a lamellar world instance

§Creating and executing a Registered Active Message

§Creating, initializing, and iterating through a distributed array

§Utilizing a Darc within an active message

§Using Lamellar

§Running Lamellar Applications

§local / shmem (single system, one or many processes)

§distributed (multi-process, multi-system)

Re-exports§

Re-exports§

Modules§

Macros§

Structs§

Enums§

Constants§

Traits§

Functions§

Attribute Macros§

Derive Macros§

Crate lamellar