Module sea_streamer_file::format
source · Expand description
The SeaStreamer file format is a container format designed to be seekable. It does not concerns what format the payload is encoded in. It has internal checksum to ensure integrity. It is a binary file format, but is readable with a plain text editor (if the payload is UTF-8).
There is a header. Every N bytes there will be a Beacon summarizing the streams so far. A message can be spliced by one or more beacons.
ⓘ
+-----------------~-----------------+
| Header |
+-----------------~-----------------+
| Message |
+-----------------~-----------------+
| Message ... |
+-----------------~-----------------+
| Beacon |
+-----------------~-----------------+
| Message ... |
+-----------------~-----------------+
Header is:
+--------+--------+---------+---~---+----~----+------+
| 0x53 | 0x73 | version | meta | padding | 0x0D |
+--------+--------+---------+---~---+----~----+------+
Header meta v1 is always 128 - 3 bytes long. Padding is stuffed with 0, ending with a \n.
Message is:
+---~----+---+----+---+----+----~----+-----+----+------+
| header | size of payload | payload | checksum | 0x0D |
+---~----+---+----+---+----+----~----+-----+----+------+
Message spliced:
+----~----+----~---+--------~-------+
| message | beacon | message cont'd |
+----~----+----~---+--------~-------+
Beacon is:
+------+-----+------+-----+------+--------------+----~----+-----+------+
| 0x0D | remaining message bytes | num of items | item | ... | 0x0D |
+------+-----+------+-----+------+--------------+----~----+-----+------+
Message header is same as beacon item:
+-------------------+--------~---------+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| len of stream key | stream key chars | shard id | seq no | timestamp |
+-------------------+--------~---------+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Except that beacon item has an extra tail:
+---------+--------+
| running checksum |
+---------+--------+All numbers are encoded in big endian. There are 0x0D in places so that it will not blow up plain text editors. And it’s semi-human-readable.
A SeaStreamer file can be terminated by a End-of-Stream Message,
with the stream key SEA_STREAMER_INTERNAL and payload EOS.
Structs§
- Timestamp in seconds