Crate flowmium

Expand description

Flowmium is a workflow orchestrator that uses Kubernetes. Flowmium is generally deployed as a binary and workflows are defined in YAML or Python and it also exposes an HTTP API. This documentation is for those who are interested in the client library or integrating flowmium into an existing Rust project instead of using the HTTP API.

Getting started

Integrating into an existing Rust project

A simple example that creates runs a workflow with a single task that prints Hello world.

use flowmium::driver;
use flowmium::executor;
use flowmium::scheduler;
use flowmium::secrets;

use flowmium::model::*;

#[tokio::main]
async fn main() {
    let subscriber = tracing_subscriber::FmtSubscriber::new();
    tracing::subscriber::set_global_default(subscriber).unwrap();

    let pool = driver::get_default_postgres_pool().await.unwrap();

    let executor_config = driver::get_default_executor_config().await.unwrap();

    let scheduler = scheduler::Scheduler::new(pool.clone());

    let secrets = secrets::SecretsCrud::new(pool.clone());
    secrets
        .create_secret("super-secret-message", "hello world")
        .await
        .unwrap();

    let handle = driver::spawn_executor(&pool, &scheduler, &executor_config);

    let flow = create_example_flow();
    executor::instantiate_flow(flow, &scheduler).await.unwrap();

    handle.await.unwrap();
}

fn create_example_flow() -> Flow {
    Flow {
        name: "hello-world".to_string(),
        tasks: vec![Task {
            name: "hello-world".to_string(),
            image: "debian:latest".to_string(),
            depends: vec![],
            cmd: vec![
                "sh".to_string(),
                "-c".to_string(),
                "echo $MESSAGE".to_string(),
            ],
            env: vec![EnvVar::SecretRef(SecretRef {
                name: "MESSAGE".to_string(),
                from_secret: "super-secret-message".to_string(),
            })],
            inputs: None,
            outputs: None,
        }],
    }
}

To run the example, you will have to run something like below

export FLOWMIUM_POSTGRES_URL='postgres://flowmium:flowmium@localhost/flowmium'
export FLOWMIUM_STORE_URL='http://localhost:9000'
export FLOWMIUM_TASK_STORE_URL='http://172.16.238.4:9000'
export FLOWMIUM_BUCKET_NAME='flowmium-test'
export FLOWMIUM_ACCESS_KEY='minio'
export FLOWMIUM_SECRET_KEY='password'
export FLOWMIUM_INIT_CONTAINER_IMAGE='docker.io/shnoo28/flowmium:latest'
export FLOWMIUM_NAMESPACE=default
export KUBECONFIG=./kubeconfig.yaml
cargo run

Crate flowmium

Getting started

Integrating into an existing Rust project

Modules