Module runtime 

Utility for managing the routing of distributed tasks and their results.

This module provides utilities for orchestrating distributed task execution, focusing on the interplay between operations, their invocations as tasks, and higher-order directives that manage these tasks.

It provides two runtimes:

• Runtime: Used by the orchestrator to manage the execution of tasks.
• WorkerRuntime: Used by worker nodes to execute tasks.

Semantic Overview

• Operation: Defines the signature and semantics of a computation. Akin to a function that maps an input to an output.
• Task: Represents the invocation of an operation with specific arguments. The payloads used for coordination.
• Directive: Manages and orchestrates the results of multiple tasks, implementing higher-order semantics.

In the context of our system, we assume:

• Operations are expensive as they usually encode hefty computations.
• Directives are cheap since they primarily manage and coordinate the results of those computations.

A key concept is the information asymmetry between Directives and Tasks:

• A Directive knows about the Tasks it manages.
• A Task, however, is oblivious to any Directive overseeing it.

This asymmetry highlights distinct channeling requirements:

• Directives require a discrete homogenous stream of results.
• Task channels must accommodate arbitrary operations, listening on a single, heterogenous stream of results.

Practically, this is implemented by using a single, stable channel for Tasks, and dynamically issuing new channels in context of each Directive’s evaluation. This allows workers to listen on a single channel and thus execute arbitrary Tasks, while Directives can listen on an isolated channel for their results.

The Runtime is implements the semantics of this architecture. Fundamentally, the Runtime is responsible for managing the channels that Tasks and Directives use to communicate, and provides a simple interface for interacting with these channels.

Structs

ExecutionErr

ExecutionOk

Runtime

The core of the distributed task management system.

WorkerRuntime

A runtime for worker nodes.

Enums

WorkerIpc

Inter-process messages between workers.

Constants

DEFAULT_ROUTING_KEY