Expand description
Malstrom is a framework for building distributed, stateful stream processes.
Modulesยง
- channels
- Channels for exchanging data between stream operators
- coordinator
- Coordinator
- errorhandling
- Global error handling mechanisms. Very TODO!
- keyed
- Keyed streams for logical and physical partitioning of data
- operators
- Operators for performing various operations on data in a job
- runtime
- Runtimes determine where Malstrom jobs are executed. Runtimes provide the necessary infrastructure and communication channels for Malstrom to run jobs.
- sinks
- Sinks for writing data from a Malstrom job
- snapshot
- Snapshots are periodically saved state from stateful operations. Regular snapshots allow resuming computation after failures. Snapshots can also be utilized to enable statful job upgrades
- sources
- Sources for reading data
- stream
- Streams are logical orders of operations. A Stream can be seen as a series of nodes and edges in the computation graph
- types
- Types and traits used accross JetStream
- worker
- Worker: A worker in Malstrom is the unit which is executing the operations in job. A Worker is also the unit of parellism i.e. the Malstrom runtime will create as many identical workers as the configured parallelism requires.