Crate sea_streamer_redis

Expand description

§`sea-streamer-redis`: Redis Backend

This is the Redis backend implementation for SeaStreamer. This crate provides a high-level async API on top of Redis that makes working with Redis Streams fool-proof:

Implements the familiar SeaStreamer abstract interface
A comprehensive type system that guides/restricts you with the API
High-level API, so you don’t call XADD, XREAD or XACK anymore
Mutex-free implementation: concurrency achieved by message passing
Pipelined XADD and paged XREAD, with a throughput in the realm of 100k messages per second

While we’d like to provide a Kafka-like client experience, there are some fundamental differences between Redis and Kafka:

In Redis sequence numbers are not contiguous
1. In Kafka sequence numbers are contiguous
In Redis messages are dispatched to consumers among group members in a first-ask-first-served manner, which leads to the next point
1. In Kafka consumer <-> shard is 1 to 1 in a consumer group
In Redis ACK has to be done per message
1. In Kafka only 1 Ack (read-up-to) is needed for a series of reads

What’s already implemented:

RealTime mode with AutoStreamReset
Resumable mode with auto-ack and/or auto-commit
LoadBalanced mode with failover behaviour
Seek/rewind to point in time
Basic stream sharding: split a stream into multiple sub-streams

It’s best to look through the tests for an illustration of the different streaming behaviour.

How SeaStreamer offers better concurrency?

Consider the following simple stream processor:

loop {
    let input = XREAD.await;
    let output = process(input).await;
    XADD(output).await;
}

When it’s reading or writing, it’s not processing. So it’s wasting time idle and reading messages with a higher delay, which in turn limits the throughput. In addition, the ideal batch size for reads may not be the ideal batch size for writes.

With SeaStreamer, the read and write loops are separated from your process loop, so they can all happen in parallel (async in Rust is multi-threaded, so it is truely parallel)!

If you are reading from a consumer group, you also have to consider when to ACK and how many ACKs to batch in one request. SeaStreamer can commit in the background on a regular interval, or you can commit asynchronously without blocking your process loop.

In the future, we’d like to support Redis Cluster, because sharding without clustering is not very useful. Right now it’s pretty much a work-in-progress. It’s quite a difficult task, because clients have to take responsibility when working with a cluster. In Redis, shards and nodes is a M-N mapping - shards can be moved among nodes at any time. It makes testing much more difficult. Let us know if you’d like to help!

You can quickly start a Redis instance via Docker:

docker run -d --rm --name redis -p 6379:6379 redis

There is also a small utility to dump Redis Streams messages into a SeaStreamer file.

This crate is built on top of redis.

Modules§

constants: More constants used throughout SeaStreamer Redis.

Structs§

Connection: A wrapped redis::aio::MultiplexedConnection that can auto-reconnect.
NextFuture
PseudoRandomSharder: Shard streams pseudo-randomly but fairly. Basically a rand() % num_shards.
RedisCluster: A set of connections maintained to a Redis Cluster with key cache.
RedisConnectOptions: Options for connections, including credentials.
RedisConsumer: The Redis Consumer.
RedisConsumerOptions: Options for Consumers, including mode, group and other streaming mechanisms.
RedisMessageStream
RedisProducer: The Redis Producer.
RedisProducerOptions: Options for Producers, including sharding.
RedisStreamer: The Redis Streamer, from which you can create Producers and Consumers.
RoundRobinSharder: Shard streams by round-robin.
SendFuture: A future that returns a Send Receipt. This future is cancel safe.

Enums§

AutoCommit: The auto ack / commit mechanism.
AutoStreamReset: Where to start streaming from if there is no priori state.
RedisErr: Different types of Redis errors.
ShardOwnership: The shard ownership model.
TimestampFormat

Constants§

DEFAULT_TIMEOUT: The default timeout, if needed but unspecified
MAX_MSG_ID: To indicate $, aka latest.
MSG: The field of the message payload
REDIS_PORT: The default Redis port number
ZERO: Shard 0

Traits§

RedisMessageId: A trait that adds some methods to RedisMessage.
Sharder: Trait that sharding strategies should implement. It should also impl Debug so its states can be inspected.
SharderConfig: Trait to instantiate new sharders. It should also impl Debug so it can be named.

Functions§

consumer_id: Generate a new consumer id, which should never collide.
group_id: Generate a new group id which should uniquely identify this host.
host_id: Get the host id. Inside Docker, it’s the container ID. Otherwise it’s the MAC address. You can override it via the HOST_ID env var.
parse_message_id: The Redis message id comprises two 64 bit integers. In order to fit it into 64 bit, we only allocate 48 bit to the timestamp, and the remaining 16 bit to the sub-sequence number.

Type Aliases§

MessageId: ID of a message in the form of (timestamp, sequence).
NodeId: ID of a node in a Redis Cluster.
RedisMessage
RedisResult: A type alias for convenience.

Crate sea_streamer_redisCopy item path

§sea-streamer-redis: Redis Backend