Module groups 

Consumer groups and the pending entries list (08 section 7).

A consumer group is a bookmark plus a ledger. The bookmark is one ID saying how far the group has read, and the ledger is every entry the group handed out and has not been told is finished with. Redis calls the ledger the PEL, the pending entries list, and an entry in it a NACK.

group "workers"  last 990-0  read 41823
+----------------------------------------------------------+
| PEL, in ID order                                          |
| 971-0 -> owner c2, handed out 3 times, last at 09:14:02   |
| 984-0 -> owner c1, handed out 1 time,  last at 09:14:07   |
| 990-0 -> owner c1, handed out 1 time,  last at 09:14:07   |
+----------------------------------------------------------+
    consumer c1 holds 984-0, 990-0
    consumer c2 holds 971-0

The point of the ledger is that a consumer can die holding work. XPENDING finds entries nobody has touched in a while and XCLAIM moves them to a consumer that is still alive, which is the whole reason to use a group rather than plain XREAD.

A NACK is in two indexes and owned by neither

Every pending entry has to be reachable two ways. XPENDING and XAUTOCLAIM walk the group’s entries in ID order, and XINFO CONSUMERS and consumer deletion need everything one consumer holds. Redis keeps a rax per group and a rax per consumer holding pointers to the same NACK, which means a claim updates a pointer in two trees and the NACK belongs to whichever one frees it last.

Here the NACK lives in the group’s map and the consumer holds only IDs. A claim moves an ID between two BTreeSets and rewrites one field, nothing is shared and nothing has to be freed carefully. It costs one extra lookup when going from a consumer’s ID to its NACK, which happens on consumer deletion and nowhere on a hot path.

Why a B-tree and not a sorted deque

The log next door is a sorted deque because entries are appended in order and trimmed from the front, and never touched in the middle. A PEL is the same shape most of the time: XREADGROUP > appends increasing IDs and XACK usually takes the oldest. But XCLAIM and a slow consumer both put holes in the middle, and an ack of an arbitrary ID is a normal thing to do rather than a pathology, so the middle is not the rare case here that it is in the log.

A BTreeMap keyed by Id holds the key inline in sixteen bytes with no allocation per entry and about eleven entries a node, and it gives the ordered walk XPENDING and XAUTOCLAIM need. That is already well ahead of a rax over sixteen byte string keys. Whether the sorted deque with tombstones would beat it is a real question and the benchmark is there to answer it, but it is not worth guessing at before the feature works.

Consumers are a vector

A group has a handful of consumers, usually as many as there are processes, and a name is looked up once per command rather than once per entry. A linear scan over a vector beats a hash map at that size and brings no dependency and no hashing with it. A slot is never reused while the group lives, so the index a NACK holds stays valid.

Structs

Consumer

One consumer inside a group.

Filter

Which pending entries a caller wants, which is every filter XPENDING takes.

Group

A consumer group over one stream.

Nack

One entry handed out and not yet acknowledged.

Enums

Retry

What releasing an entry does to its delivery count.