seqlog 0.1.0

An embedded append-only sequence log store.
Documentation
An embedded append-only sequence log store.

SeqLog is designed for workloads where records are identified by monotonically
increasing sequence numbers and are primarily accessed in sequence order.
Typical use cases include event storage, write-ahead logs (WAL), market data
capture, message persistence, and other append-heavy workloads.

SeqLog can be viewed as a specialized key-value store, where the Key is
a monotonically increasing sequence number (`u64`), and the Value is
arbitrary bytes (`[u8]`).


# Features

- Append-only writes,
- Sequence-based seeking and sequential reads,
- Truncation from either the head or tail,
- No in-place updates;

and

- High performance and throughput. The benchmark is in processing.
- Predictable and compact storage layout. The overhead (including index)
is less 1% in most cases. See the [Storage Layout]#storage-layout section
for detail.
- No crate dependencies, although this is not a goal in itself.


# Usage

```rust
fn example() -> std::io::Result<()> {
    // Create a SeqLog instance, with sequence number starting from 100.
    // Use `open()` method to open an existing SeqLog instance.
    let mut store = seqlog::SeqLog::create("target/example-store/", 100)?;

    // Append entries in batch.
    let entries = vec![
        "Hello, world!",
        "The value could be arbitrary bytes.",
        "Live & Learn!",
    ];
    store.append(&entries)?;

    // Assigned sequence numbers:
    // 100 -> "Hello, world!"
    // 101 -> "The value could be arbitrary bytes."
    // 102 -> "Live & Learn!"
    assert_eq!(store.next_seq(), 103);

    // Create a reader to read sequentially, from sequence number 102.
    // You can create multiple readers, with independent cursors.
    let mut reader = store.reader(102)?;
    assert_eq!(reader.next()?, Some(entries[2].as_bytes())); // read 102
    assert_eq!(reader.next()?, None); // EOF
    Ok(())
}

// example().unwrap();
```


# Storage Layout

This section describes implementation details that are not required for
normal use of SeqLog.

A SeqLog instance is stored as a directory:

```bash
$ ls target/example-store/
00000001.seqlog LOCK
```

There is one `LOCK` file and one or more `.seqlog` data files.
Data files rotate automatically by size or manually via API. 

There is no separate index file. Instead, SeqLog uses a sparse index embedded
in the data files.

A data file consists of multiple fixed-size (64KiB) blocks.

A block contains a starting sequence number (8 bytes), followed by multiple
entries and optional padding.

An entry consists of a length (2 bytes) followed by its data.

The block format is shown as:

```text
+8--------+
|start-seq|
+2--+-----+---------+
|len| entry         |
+2--+-------------------------+
|len| entry                   |
+---+-------------------------+
:
:
+2--+-------------------+
|len| entry             |
+?------+---------------+
|padding|
+-------+
```

The following example shows a real data file generated by the Usage example.
Since this is the final block in the file, there is no padding.

```bash
$ hexdump -C target/example-store/00000001.seqlog
00000000  64=00=00=00=00=00=00=00  0d-00 48 65 6c 6c 6f 2c  |d.........Hello,|
00000010  20 77 6f 72 6c 64 21 23--00 54 68 65 20 76 61 6c  | world!#.The val|
00000020  75 65 20 63 6f 75 6c 64  20 62 65 20 61 72 62 69  |ue could be arbi|
00000030  74 72 61 72 79 20 62 79  74 65 73 2e 0d-00 4c 69  |trary bytes...Li|
00000040  76 65 20 26 20 4c 65 61  72 6e 21                 |ve & Learn!|
```

Each block stores its starting sequence number, which serves as a sparse index.
As a result, the indexing overhead is extremely small.

Writes are append-only at both the entry level and the file level. No in-place
updates are required, allowing write operations to remain simple and efficient.

Reading is optimized for sequential workloads rather than random access. To
locate a sequence number, SeqLog first finds the target file using the
starting sequence number recorded for each file in memory, then locates the
corresponding block, and finally performs a sequential scan within that block.
This makes seeking slower than in fully indexed storage engines. In practice,
however, seeks are relatively infrequent in append-heavy log-processing
workloads, while the sparse-index design keeps both storage overhead and write
amplification low.


# design (TODO)

- one big fixed-size file vs rotated files
- why no memmap
- why no compress
- why no encoding/ser
- why no timestamp
- why no multi-write
- why no crc32


# comparison (TODO)

- commitlog
- seglog
- walcraft
- varvedb
- factstr