Crate hopper [] [src]

hopper - an unbounded mpsc with bounded memory

This module provides a version of the rust standard mpsc that is unbounded but consumes a bounded amount of memory. This is done by paging elements to disk at need. The ambition here is to support mpsc style communication without allocating unbounded amounts of memory or dropping inputs on the floor.

Hopper is intended to be used in situtations where your system cannot load-shed inputs and must eventually process them. Hopper does page to disk but has the same durabilty guarantees as stdlib mpsc between restarts: none.

Inside Baseball

Hopper's channel looks very much like a named pipe in Unix. You supply a name to either channel_2 or channel_with_max_bytes_3 and you push bytes in and out. The disk paging adds a complication. The name supplied to the above two functions is used to create a directory under user-supplied data_dir. This directory gets filled up with monotonically increasing files.

The on-disk structure look like so:


You'll notice exports of Sender and Receiver in this module's namespace. These are the structures that back the send and receive side of the named channel. The Senders--there may be multiples of them--are responsible for creating "queue files". In the above, data-dir/sink-name*/* are queue files. These files are treated as append-only logs by the Senders. The Receivers trawl through these logs to read the data serialized there.

Won't this fill up my disk?

Maybe! Each Sender has a notion of the maximum bytes that a queue file may consume--which you can set explicitly when creating a channel with channel_with_explicit_capacity--and once the Sender has gone over that limit it'll attempt to mark the queue file as read-only and create a new file. The Receiver is programmed to read its current queue file until it reaches EOF and, finding the file is read-only, removes the queue file and moves on to the next.

If the Receiver is unable to keep up with the Senders then, oops, your disk will gradually fill up.

What kind of filesystem options will I need?

Hopper is intended to work on any wacky old filesystem with any options, even at high concurrency. As common filesystems do not support interleaving small atomic writes hopper limits itself to one exclusive Sender or one exclusive Receiver at a time. This potentially limits the concurrency but maintains data integrity. We are open to improvements in this area.



The 'receive' side of hopper, similar to std::sync::mpsc::Receiver.


The 'send' side of hopper, similar to std::sync::mpsc::Sender.



Defines the errors that hopper will bubble up



Create a (Sender, Reciever) pair in a like fashion to std::sync::mpsc::channel


Create a (Sender, Reciever) pair in a like fashion to std::sync::mpsc::channel