Crate selium_log

A library containing an implementation of an ordered, log-based message queue.

Selium Log aims to provide as simple an abstraction as possible over the message log, in order to make it easy to provision and use in your libraries.

The message log’s structure should be familiar to those coming from an Apache Kafka background: a log is represented by one or more segments, with each segment comprised of a memory-mapped index file, serving as a lookup for the segment’s records, and an append-only data file containing said records.

Only the most current segment can be designated as the mutable (“hot”) segment, while the older segments are read-only until their eventual cleanup. Once the current hot segment exceeds the defined LogConfig::max_index_entries threshold, it will also become read-only, while a new segment is created and assigned as the hot segment.

Replication has not yet been implemented for Selium Log as of this release, but is a planned feature. Due to this, durability can be tough to achieve alongside high throughput on a single node. Most of the latency comes from the I/O overhead of flushing the memory-mapped index and data files to the filesystem. To compensate for this overhead, the flushing frequency can be tweaked via the FlushPolicy struct, to strike a balance between durability and throughput.

Modules

config

Configuration settings for an individual message log.

data

Contains the Data type, along with a LogIterator to iterate over the data file.

error

Type aliases and Enums relating to Selium Log errors.

index

Contains the Index and Mmap types.

message

Message envelope/frame for a set of records.

segment

Contains the Segment and SegmentList types.

Structs

MessageLog

The entry point to Selium Log.