seqlog 0.1.0

An embedded append-only sequence log store.
Documentation
  • Coverage
  • 94.44%
    17 out of 18 items documented1 out of 17 items with examples
  • Size
  • Source code size: 32.1 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 512.5 kB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 4s Average build duration of successful builds.
  • all releases: 5s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • WuBingzheng/seqlog
    0 0 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • WuBingzheng

An embedded append-only sequence log store.

SeqLog is designed for workloads where records are identified by monotonically increasing sequence numbers and are primarily accessed in sequence order. Typical use cases include event storage, write-ahead logs (WAL), market data capture, message persistence, and other append-heavy workloads.

SeqLog can be viewed as a specialized key-value store, where the Key is a monotonically increasing sequence number (u64), and the Value is arbitrary bytes ([u8]).

Features

  • Append-only writes,
  • Sequence-based seeking and sequential reads,
  • Truncation from either the head or tail,
  • No in-place updates;

and

  • High performance and throughput. The benchmark is in processing.
  • Predictable and compact storage layout. The overhead (including index) is less 1% in most cases. See the Storage Layout section for detail.
  • No crate dependencies, although this is not a goal in itself.

Usage

fn example() -> std::io::Result<()> {
    // Create a SeqLog instance, with sequence number starting from 100.
    // Use `open()` method to open an existing SeqLog instance.
    let mut store = seqlog::SeqLog::create("target/example-store/", 100)?;

    // Append entries in batch.
    let entries = vec![
        "Hello, world!",
        "The value could be arbitrary bytes.",
        "Live & Learn!",
    ];
    store.append(&entries)?;

    // Assigned sequence numbers:
    // 100 -> "Hello, world!"
    // 101 -> "The value could be arbitrary bytes."
    // 102 -> "Live & Learn!"
    assert_eq!(store.next_seq(), 103);

    // Create a reader to read sequentially, from sequence number 102.
    // You can create multiple readers, with independent cursors.
    let mut reader = store.reader(102)?;
    assert_eq!(reader.next()?, Some(entries[2].as_bytes())); // read 102
    assert_eq!(reader.next()?, None); // EOF
    Ok(())
}

// example().unwrap();

Storage Layout

This section describes implementation details that are not required for normal use of SeqLog.

A SeqLog instance is stored as a directory:

$ ls target/example-store/
00000001.seqlog LOCK

There is one LOCK file and one or more .seqlog data files. Data files rotate automatically by size or manually via API.

There is no separate index file. Instead, SeqLog uses a sparse index embedded in the data files.

A data file consists of multiple fixed-size (64KiB) blocks.

A block contains a starting sequence number (8 bytes), followed by multiple entries and optional padding.

An entry consists of a length (2 bytes) followed by its data.

The block format is shown as:

+8--------+
|start-seq|
+2--+-----+---------+
|len| entry         |
+2--+-------------------------+
|len| entry                   |
+---+-------------------------+
:
:
+2--+-------------------+
|len| entry             |
+?------+---------------+
|padding|
+-------+

The following example shows a real data file generated by the Usage example. Since this is the final block in the file, there is no padding.

$ hexdump -C target/example-store/00000001.seqlog
00000000  64=00=00=00=00=00=00=00  0d-00 48 65 6c 6c 6f 2c  |d.........Hello,|
00000010  20 77 6f 72 6c 64 21 23--00 54 68 65 20 76 61 6c  | world!#.The val|
00000020  75 65 20 63 6f 75 6c 64  20 62 65 20 61 72 62 69  |ue could be arbi|
00000030  74 72 61 72 79 20 62 79  74 65 73 2e 0d-00 4c 69  |trary bytes...Li|
00000040  76 65 20 26 20 4c 65 61  72 6e 21                 |ve & Learn!|

Each block stores its starting sequence number, which serves as a sparse index. As a result, the indexing overhead is extremely small.

Writes are append-only at both the entry level and the file level. No in-place updates are required, allowing write operations to remain simple and efficient.

Reading is optimized for sequential workloads rather than random access. To locate a sequence number, SeqLog first finds the target file using the starting sequence number recorded for each file in memory, then locates the corresponding block, and finally performs a sequential scan within that block. This makes seeking slower than in fully indexed storage engines. In practice, however, seeks are relatively infrequent in append-heavy log-processing workloads, while the sparse-index design keeps both storage overhead and write amplification low.

design (TODO)

  • one big fixed-size file vs rotated files
  • why no memmap
  • why no compress
  • why no encoding/ser
  • why no timestamp
  • why no multi-write
  • why no crc32

comparison (TODO)

  • commitlog
  • seglog
  • walcraft
  • varvedb
  • factstr