Crate sea_streamer_file

Expand description

§`sea-streamer-file`: File Backend

This is very similar to sea-streamer-stdio, but the difference is SeaStreamerStdio works in real-time, while sea-streamer-file works in real-time and replay. That means, SeaStreamerFile has the ability to traverse a .ss (sea-stream) file and seek/rewind to a particular timestamp/offset.

In addition, Stdio can only work with UTF-8 text data, while File is able to work with binary data. In Stdio, there is only one Streamer per process. In File, there can be multiple independent Streamers in the same process. Afterall, a Streamer is just a file.

The basic idea of SeaStreamerFile is like a tail -f with one message per line, with a custom message frame carrying binary payloads. The interesting part is, in SeaStreamer, we do not use delimiters to separate messages. This removes the overhead of encoding/decoding message payloads. But it added some complexity to the file format.

The SeaStreamerFile format is designed for efficient fast-forward and seeking. This is enabled by placing an array of Beacons at fixed interval in the file. A Beacon contains a summary of the streams, so it acts like an inplace index. It also allows readers to align with the message boundaries. To learn more about the file format, read src/format.rs.

On top of that, are the high-level SeaStreamer multi-producer, multi-consumer stream semantics, resembling the behaviour of other SeaStreamer backends. In particular, the load-balancing behaviour is same as Stdio, i.e. round-robin.

§Decoder

We provide a small utility to decode .ss files:

cargo install sea-streamer-file --features=executables --bin ss-decode
 # local version
alias ss-decode='cargo run --package sea-streamer-file --features=executables --bin ss-decode'
ss-decode -- --file <file> --format <format>

Pro tip: pipe it to less for pagination

ss-decode --file mystream.ss | less

Example log format:

 # header
[2023-06-05T13:55:53.001 | hello | 1 | 0] message-1
 # beacon

Example ndjson format:

/* header */
{"header":{"stream_key":"hello","shard_id":0,"sequence":1,"timestamp":"2023-06-05T13:55:53.001"},"payload":"message-1"}
/* beacon */

There is also a Typescript implementation under sea-streamer-file-reader.

§TODO

Resumable: currently unimplemented. A potential implementation might be to commit into a local SQLite database.
Sharding: currently it only streams to Shard ZERO.
Verify: a utility program to verify and repair SeaStreamer binary file.

Modules§

export
format: The SeaStreamer file format is a container format designed to be seekable. It does not concerns what format the payload is encoded in. It has internal checksum to ensure integrity. It is a binary file format, but is readable with a plain text editor (if the payload is UTF-8).

Structs§

AsyncFile: A minimal wrapper over async runtime’s File.
ByteBuffer: A FIFO queue of Bytes.
FileConnectOptions
FileConsumer
FileConsumerOptions
FileId: Basically a file path.
FileMessageStream
FileProducer
FileProducerOptions
FileReader: A simple buffered and bounded file reader. The implementation is much simpler than FileSource.
FileSink: Buffered file writer.
FileSinkWriter: A delegate that impl std::io::Write
FileSource: FileSource treats files as a live stream of bytes. It will read til the end, and will resume reading when the file grows. It relies on notify::RecommendedWatcher, which is the OS’s native notify mechanism. The async API allows you to request how many bytes you need, and it will wait for those bytes to come in a non-blocking fashion.
FileSourceFuture
FileStreamer
MessageSink: A high level file writer that mux messages and beacon
MessageSource: A high level file reader that demux messages and beacon
MockBeacon
SendFuture
Surveyor: The goal of Surveyor is to find the two closest Beacons that pince our search target. If would be pretty simple, if not for the fact that a given location may not contain a relevant Beacon, which could yield Undecided.

Enums§

AutoStreamReset: Where to start streaming from.
Bytes: A blob of bytes; optimized over byte and word.
ConfigErr
DynFileSource: A runtime adapter of FileReader and FileSource, also able to switch between the two mode of operations dynamically.
DynReadFuture
FileErr
FileSourceType
NextFuture
ReadFrom
SeekErr
SeekTarget
StreamMode
SurveyResult

Constants§

DEFAULT_BEACON_INTERVAL
DEFAULT_FILE_SIZE_LIMIT
DEFAULT_PREFETCH_MESSAGE
END_OF_STREAM
PULSE_MESSAGE
SEA_STREAMER_WILDCARD: Reserved by SeaStreamer. Avoid using this as StreamKey.

Traits§

Appendable
BeaconReader
ByteSink
ByteSource
HasMessageHeader

Functions§

end_of_stream: This can be written to a file to properly end the stream
is_end_of_stream
query_streamer: Query info about global Streamer(s) topology

Type Aliases§

FileMessage
FileReaderFuture
FileResult

Crate sea_streamer_fileCopy item path

§sea-streamer-file: File Backend