Broker constitutes of following components
----------------
* N connections. 1 per client
* Router which holds all subscription information and handles to
communicate with all the clients. This might not be super efficient as
data travels from connection 1 -> router -> connection 2. But might
not be significant if prevent clones (just send fat pointers) and batch
* Same router which reads device state from all the connections and actions
from the backend
* Timestone which reads all the data from all the connections and writes
to disk and tries to simulate kafka
Device id as part of topic in subscriptions and publishes
-----------------
* Maybe useful to directly send publishes to backend (where router picks up device id directly)
* Subscriptions only receive data directed to it
* Get in the way of wildcards inplace of device ids
Broker cluster and replication
-----------------
* State sits with connection at this time. Connection receives publishes
acks and forwards data to router.
* This might get in the way of HA and replication as connection acks
without not being sure if router has replicated the data
Current design:
connection 1 -> router -> connection 2
| |
ack replication
Alternate:
connection 1 -> router -> connection 2
|
ack <- replication
But this puts the overhead of sending all acks back to the connection
over a channel and receiving acks of forwards to update the state. Maybe
we can microbatch smartly.
Advantages of router maintaining the state
---------------
* Necessary for replication to happen centrally. Router maintains
connections to all the other brokers with out each connection
maintaining these. Router replicates and sends ack to the connection
where it's written to the network
* All distributed logic at one place
* Connections are stateless
Disadvantages
-------------
* Acks from/to router. Adds to processing. Microbatching can help here
Shared router experiment
-------------
* Router maintains all the state of connections
Connections just forward data to the router
* Router can be a shared with multiple connections on a thread with Rc
This can make the router forwarding asynchronous
Having a router as a separate thread makes asynchronous forwarding
difficult as number of connections are dynamic and we need to `select`
over them
* 4 threaded broker will have 4 routers which caters n connections
This is a good middle ground between each connection having knowledge of
all the other connections vs single router thread maintaining all the
connection knowledge.
Each router now has info about all it's connections and the other
routers
* Routers will talk to each other to forward publisher to a connection
on different router
We don't want to forward data directly to the connection from a random
router as we want to save the outgoing state of a connection on a
single router. Prevents fragmentation
* This inter router communication can be extended across network to make
the broker distributed
Findings:
-------
* Need to separate mutable and immutable parts well
* Can't use tokio channel as sends need mutable borrows
* Even async std channels are a problem because the router can block
during `rc_router_tx.send()` causing Borrow mut panic while creating a
new connection
* Difficult to see Bandwidth of router and connections separately
Distributed commit log
----------------
* Router design should be extend to be a distributed commit log like
kafka
* But doing disk operations on router (which is shared by async
connections) is not correct
We probably can use BufWriter to fill the file and forward it to a
different thread?
* We are now copying the outoing state of each connection. We need to
save a single commitlog per topic make make the subscribers just hold
offset like kafka does
TODO
---------------
* Use hash of connection id strings to numbers and validate perf
* Make sure that whole router doesn't slowdown because of one slow connection
* Inflight message limits. This should be similar to client. Slow acks should
throttle down the connection eventloop channel receiver. This
pushes the backpressure to the router. But this should not slow down
other connections. Essentially everything should be concurrent in a
select! in the router
* Add throughput metrics to every connection. Logging based
instrumentation will be very handy here but we don't have a design on
dashboards on top of log based metrics
* Production ready accept loop. https://github.com/async-rs/async-std/pull/666/files
* Better errors when tls connection happens on tcp port
```
Client side error:
Received = StreamEnd(Network(Io(Custom { kind: UnexpectedEof, error: "tls handshake eof" })))
Server side error:
ERROR librumqd::connection > Connect packet error = Timeout(Elapsed(()))
```
References
--------------
* https://bulldog2011.github.io/blog/2013/03/27/the-architecture-and-design-of-a-pub-sub-messaging-system/
