Expand description
Durable NATS JetStream KV store for nopace.
§What one key holds
One key holds the state of one partition inside one scope, and nothing else. That is the whole design, and it is worth being explicit about why, because the obvious alternative — one key per scope — is what this crate used to do and it took a service down for nine and a half hours.
A scope is the budget’s identity, so applications reasonably configure one
constant scope and put the customer in the partition. Serializing a scope
meant serializing every partition in it into a single KV value, so that
value grew linearly with the tenant count. At about two thousand tenants it
crossed the server’s max_payload, NATS refused the write, and because the
write error was discarded the store retried the same oversized value for
ever. Every admission blocked, every request failed closed, and no counter
moved. Keying by partition makes the value’s size a function of one
tenant’s traffic instead of the whole customer base, and it is also what
stops unrelated tenants from contending on one revision.
§Delete is a reset
Deleting or purging a KV key does not remove it; it appends a tombstone,
which Store::entry returns with an empty value and a real revision. The
previous implementation fed that empty value to the deserializer and
returned “invalid state” for ever after, so the documented operator
mitigation — nats kv del — bricked the key instead of resetting it. The
same went for writing an empty value by hand, which is an ordinary put and
not a tombstone at all. Anything empty is now read as absent state, while
its revision is still used for the next compare-and-set, so a reset is a
reset and the write that follows it is still atomic.
Bytes that are present but do not decode stay an error: that is corruption or an incompatible build, not a reset, and quietly discarding live quota for it would be worse than refusing.
§What this backend cannot do
All claims of one call are admitted atomically. Two partitions are two
keys, and two KV keys have no shared transaction, so a call whose claims
span more than one partition is refused with
NatsError::CrossPartitionCall rather than silently split into
independent decisions that would each let the other’s traffic through.
Model such a call as one partition — the group’s identity — or use a
backend whose durable unit is the scope.