An embedded append-only sequence log store.
SeqLog is designed for workloads where records are identified by monotonically increasing sequence numbers and are primarily accessed in sequence order. Typical use cases include event storage, write-ahead logs (WAL), market data capture, message persistence, and other append-heavy workloads.
SeqLog can be viewed as a specialized key-value store, where the Key is
a monotonically increasing sequence number (u64), and the Value is
arbitrary bytes ([u8]).
Features
- Append-only writes,
- Sequence-based seeking and sequential reads,
- Truncation from either the head or tail,
- No in-place updates;
and
- High performance and throughput. The benchmark is in processing.
- Predictable and compact storage layout. The overhead (including index) is less 1% in most cases. See the Storage Layout section for detail.
- No crate dependencies, although this is not a goal in itself.
Usage
// example().unwrap();
Storage Layout
This section describes implementation details that are not required for normal use of SeqLog.
A SeqLog instance is stored as a directory:
There is one LOCK file and one or more .seqlog data files.
Data files rotate automatically by size or manually via API.
There is no separate index file. Instead, SeqLog uses a sparse index embedded in the data files.
A data file consists of multiple fixed-size (64KiB) blocks.
A block contains a starting sequence number (8 bytes), followed by multiple entries and optional padding.
An entry consists of a length (2 bytes) followed by its data.
The block format is shown as:
+8--------+
|start-seq|
+2--+-----+---------+
|len| entry |
+2--+-------------------------+
|len| entry |
+---+-------------------------+
:
:
+2--+-------------------+
|len| entry |
+?------+---------------+
|padding|
+-------+
The following example shows a real data file generated by the Usage example. Since this is the final block in the file, there is no padding.
||
| |
||
||
| & |
Each block stores its starting sequence number, which serves as a sparse index. As a result, the indexing overhead is extremely small.
Writes are append-only at both the entry level and the file level. No in-place updates are required, allowing write operations to remain simple and efficient.
Reading is optimized for sequential workloads rather than random access. To locate a sequence number, SeqLog first finds the target file using the starting sequence number recorded for each file in memory, then locates the corresponding block, and finally performs a sequential scan within that block. This makes seeking slower than in fully indexed storage engines. In practice, however, seeks are relatively infrequent in append-heavy log-processing workloads, while the sparse-index design keeps both storage overhead and write amplification low.
design (TODO)
- one big fixed-size file vs rotated files
- why no memmap
- why no compress
- why no encoding/ser
- why no timestamp
- why no multi-write
- why no crc32
comparison (TODO)
- commitlog
- seglog
- walcraft
- varvedb
- factstr