protosocket

Message-oriented, low-abstraction tcp streams.

A protosocket is a non-blocking, bidirectional, message streaming connection. Providing an encoder and decoder for your messages, you can stream to and from tcp servers.

There is no wrapper encoding - no HTTP, no gRPC, no websockets. You depend on TCP (and any wrapper you want, like TLS) and your serialization strategy.

Dependencies are trim; tokio is the main heavy dependency. If you use protocol buffers, you will also depend on prost. There's no extra underlying framework like axum or hyper.

Protosocket avoids opinions - you choose your own message ordering and concurrency semantics. You can make an implicitly ordered stream, or a non-blocking out-of-order stream, or RPC request/response, or whatever.

Tools to help working with protocol buffers are provided in protosocket-prost.

You can write an RPC client/server with protosocket-rpc.

You can see an example of protocol buffers RPC in example-proto.

Case study

Background

(Full disclosure: I work at Momento at time of writing this): Momento has historically been a gRPC company. In a particular backend service with a fairly high message rate, the synchronization in h2 under tonic was seen to be blocking threads in the tokio request runtime too much. This was causing task starvation and long polls.

The starvation was tough to see, but it happens with lock_contended stacks underneath the std::sync::Mutex while trying to work with h2 buffers. That mutex is okay, but when the futex syscall parks a thread, it takes hundreds of microseconds to get the thread going again on Momento's servers. It causes extra latency that you can't easily measure, because tasks are also not picked up promptly in these cases.

I was able to get 20% greater throughput by writing k-lock and replacing the imports in h2 for std::sync::Mutex with k_lock::Mutex. This import-replacement for std::sync::Mutex tries to be more appropriate for tokio servers. Basically, it uses a couple heuristics to both wake and spin more aggressively than the standard mutex. This is better for tokio servers, because those threads absolutely must finish poll() asap, and a futex park blows the poll() budget out the window.

20% wasn't enough, and the main task threads were still getting starved. So I pulled protosocket out of rmemstore, to try it out on Momento's servers.

Test setup

Momento has a daily latency and throughput test to monitor how changes are affecting system performance. This test uses a small server setup to more easily stress the service (as opposed to stressing the load generator).

Latency is measured outside of Momento, at the client. It includes a lot of factors that are not directly under Momento control, and offers a full picture of how the service could look to users (if it were deployed in a small setup like the test).

The number Momento looked at historically was at which throughput threshold does the server pass 5 milliseconds at p99.9 tail latency?

Achievable throughput increased, and latency at all throughputs was significantly reduced. This improved the effective vertical scale of the reference workflow.

The effective vertical scale of the small reference server was improved by 2.75x for this workflow by switching the backend protocol from gRPC to protosocket.

Continuing investment

As Momento leaned progressively into protosocket, the difference grew wider. The v1.0 effort on protosocket focused on removing abstractions and supporting more extensibility in &mut self contexts. Other needs were found, like spillway, via flamegraphs.

Nowadays Momento is far past the previous gRPC benchmark. With protosocket v1 and further efforts, the old throughput/latency benchmark was too lax a standard. We now do 100khz on the same server every morning and see sub-millisecond p99.9, and sub-3ms p99.99, measured at the client. Performance per ec2 dollar, for small rpcs, is over 10 times better on protosocket v1 as compared to gRPC.

Results

	Throughput	Latency
gRPC, optimized
protosocket v0, initial integration
protosocket v1, optimized