# Stream segments: writing a checkpoint while it is produced
`BlockPtrFormat::StreamSegment` lays every object out tail first, so data can go
to storage while it is still being written instead of all at once when a flush
asks for it. A flush is left with what has not been sent yet, plus a small part 0.
The format is chosen when a container is created, like any block-pointer format.
How data is sent ahead is a runtime setting, `stream_segment_bytes`, and can differ
between sessions on the same container. Builds with and without the WAL write the
same layout, so a container can be written by either in turn; see
[With the WAL](#with-the-wal).
## Layout
```text
[data blocks][meta blocks][summary entries][header]
n × data m × meta n × 16 256 B
```
A data block's place is `k × data block size` the moment it is placed, whatever
comes after it, so its pointer is final before the object is finished. The index
nodes point at the data, so they come after it. The summary lists the data blocks.
The header comes last, because it counts everything, names the object and carries
the inode.
The header is self-contained: magic, layout version, flags, checkpoint number and
part, block sizes, block counts, the inode the checkpoint was built on, the inode,
and a CRC-32. A reader finds it in an object's last 256 bytes and refuses it unless
the magic, version and CRC hold, the counts add up to exactly the object's length,
and it names the object it was read from. A layout version newer than the build is
`Unsupported`.
Objects are named as under `PartedSegment`: `<cno>.<part>`, part 0 being the
checkpoint. A reader that knows the format reads an object's header with a
suffix request, then what it wants of the object exactly: the index nodes for a
lookup in the block map, or the summary when opening a checkpoint. The two are
about the same size, so neither is read for the other. A cold read fetches each
object's index nodes once: those the walk will need later, under nodes already
loaded, are kept from that fetch. A reader that does not know the format (opening a given checkpoint, the
cleaner) looks at the end of a part first and reads it from its start when the
end is not a stream header. An unsuffixed object is never tail first.
## Every object describes itself
Each object holds its data, the leaves of the block map over that data, a summary
naming every block, and a header. A leaf is never split from its data: the blocks
under one leaf go into one object, and that leaf goes with them. An object can
therefore be a little over its target, but never over `segment_max_object_bytes`
nor the 1 GiB a pointer addresses in one object. Part 0 holds the index nodes above the
leaves, the leaves with no new data under them, whatever data still fits, and the
inode.
A block written again after it was placed in an object leaves a dead slot there.
Its summary entry says so, the block stays dirty, and a later object or the flush
writes it again.
## Sending ahead: `stream_segment_bytes`
The target size of the objects data is sent in while it is written. It is capped
at `segment_max_object_bytes`. Default 1 GiB.
| `0` | nothing sent ahead | as many as the flush needs | all the data |
| ≤ one `segment_mpu_chunk_size` piece | each object sent whole, in one request, once there is one's worth | many small ones | the last small object |
| larger | one object filled a piece at a time and closed once it holds the target | few, about the target | the open object's last piece and its completion |
No object sent ahead is larger than the flush would make one:
`segment_max_object_bytes`, and never more than 1 GiB. Its data, the leaves over
that data, its summary and its header all count. Before each piece is placed, the
object is checked for room for it and its leaves, with one leaf in sixteen kept
spare for leaves that split before the object closes. An object without room is
closed early, and the piece starts the next one. Sent whole, an object takes what
fits and leaves the rest for the next. When the object closes, it is measured once
more. Leaves that split more than the spare allowed, from writes between its
blocks, would take it past the limit. An object like that is given up, as one that
failed is: its upload is aborted, its blocks stay dirty and are sent again, and
they count in `stream_dead_bytes`.
Sending ahead moves work onto the thread that writes: choosing the blocks,
copying a dirty block for its upload and hashing it, and setting its pointer,
which a flush would otherwise do later. Measured on memory staging, 512 MiB in
16 MiB writes at 4 KiB blocks: the writer spent 0.83 s/GiB by value, against
0.37 under `PartedSegment`, and 0.22 by reference, against 0.11.
In both ways of sending ahead, at most `segment_mpu_max_concurrency` objects or
pieces are in flight, each holding a copy of its blocks. A block leaves the dirty
tier once the object holding it is complete, if two things still hold: its
pointer is where it was sent, and a hash of its bytes matches what was sent. A
leaf whose bytes are unchanged and that has no dirty block under it becomes clean.
Nothing in a multipart upload is readable until it completes. So with objects
filled in pieces, the blocks of the open object stay in memory until it closes,
and the dirty-data limits still apply.
At a flush, whatever is still unsent goes up as pieces, several at once, and the
open object closes. Part 0 is written after every object has completed, then the
inode. The checkpoint's number is taken once. Every object and every attempt of
the flush uses it, and each attempt reserves its part numbers before writing.
An object that cannot complete is aborted. Its blocks are still dirty, so the
flush writes them and nothing is lost. This is not an error of the flush. A
panic in a task sending an object is different: it is a bug, and it is raised
again in the call that next takes the object in (a write, a flush or a
rollback), as if it had happened there. A write whose answer was lost and which the
SDK resent completes, as any conditional write does; see `x-amz-meta-hf-write` in
[concurrency](concurrency.md).
## The dirty-data limits come first
**With the default limits nothing is sent ahead.** An object can be no larger
than the dirty data the file may hold — its blocks stay in memory until it is
complete — and that is
```text
min(max(data_cache_dirty_max_bytes_threshold,
data_cache_dirty_max_blocks_threshold × data_block_size),
segment_buffer_size)
```
which is 8 MiB by default: less than one `segment_mpu_chunk_size` piece, 16 MiB.
The limit is crossed before a piece's worth is unsent, and crossing it writes a
partial segment the writer waits for, as under `PartedSegment`. Measured, 512 MiB
in 1 MiB writes with the default limits:
| directory | 5.2 s, 57 objects of ~9 MiB | 5.1 s, the same objects, nothing sent ahead | 5.1 s, the same |
| general purpose | 8.0 s | 8.1 s | 8.0 s |
against 1.2–3.4 s with the limits raised. The longest single write was 0.1–0.2 s,
waiting for a partial.
So raise the limits with the format, to at least the object size wanted plus the
pieces in flight:
```rust
rt.data_cache_dirty_max_bytes_threshold = 1 << 30; // and memory to match
rt.segment_buffer_size = 1 << 30;
rt.stream_segment_bytes = 1 << 30; // the default
```
With objects sent whole (`stream_segment_bytes` up to one piece), the limits need
only hold a few objects — `segment_mpu_max_concurrency` of them in flight, each
also copied while it uploads. With objects filled in pieces they must hold a whole
object: one closes early, at whatever size it has, when the limits are crossed.
## Blocks written by reference
Blocks written with `write_ref` are sent ahead with the dirty blocks, in the same
objects. Their bytes are read from the block source on a blocking thread when the
piece holding them is uploaded, as the flush reads them, so they are never held in
the dirty tier. The memory they cost is the pieces in flight.
The dirty-data limits do not apply to them. A reference never publishes on its own
and holds no bytes, so what sends it is only the unsent bytes reaching a piece, or
an object sent whole. With the default limits, a file written only by reference is
sent ahead.
When its object is complete, a block sent by reference is stored: it stops being a
reference, it is read from that object, and the flush does not read it from the
source again. If it was written again in the meantime, by reference, by value,
zeroed or in part, its pointer has moved and it leaves a dead slot, as a dirty
block does. A block whose object failed is still a reference, and the flush reads
it from the source.
The source's contract is unchanged: from `write_ref` until the flush that
publishes the checkpoint returns, it gives the bytes as they were at `write_ref`.
Reading them early falls within that.
Measured on a general purpose bucket: 512 MiB written by reference at about
200 MB/s, in 1 MiB calls, with the default dirty-data limits, then the flush:
| `PartedSegment` | 1.18–1.21 s |
| `StreamSegment`, nothing sent ahead | 1.11–1.23 s |
| `StreamSegment`, 1 GiB target, 16 MiB pieces | 0.41–0.46 s |
## With the WAL
A build with the WAL writes checkpoints in the same layout and reads those any
build wrote. Three paths publish, and each writes the stream layout:
- the reactor's WAL publish builds the checkpoint as one object, part 0, keeps it
pinned in memory while it uploads, and serves reads from it until it lands, as
it does under the other formats; with the WAL a checkpoint published this way is
always one object, and one larger than `segment_max_object_bytes` is refused;
- a flush on the direct API, and a replay at open, lay the checkpoint out as a
build without the WAL does, in parts when it is larger than one object;
- memory pressure inside a transaction writes partials, as elsewhere.
Nothing is sent ahead in a build with the WAL: `stream_segment_bytes` is not used
there. The reactor's WAL publish already hands the upload to the background and
lets reads and writes go on.
## What it buys
512 MiB written in 1 MiB writes at a fixed rate, then one flush.
`segment_max_object_bytes` 1 GiB, pieces 16 MiB, and the dirty-data limits raised
out of the way (see above). The reactor's handler (`fh_write`, `fh_flush`) gives
the same figures as the direct API. Flush time in seconds, with
objects per checkpoint in parentheses:
| directory | 100 MB/s | 0.74 (1) | 0.07 (129) | 0.18 (33) | 0.17 (3) | 0.18 (2) | 0.19 (2) |
| directory | 400 MB/s | 0.74 (1) | 0.08 (128) | 0.17 (33) | 0.17 (3) | 0.18 (2) | 0.19 (2) |
| general purpose | 100 MB/s | 1.22 (1) | 0.17 (129) | 0.42 (33) | 0.45 (3) | 0.43 (2) | 0.48 (2) |
| general purpose | 400 MB/s | 1.18 (1) | 0.58 (108) | 0.76 (29) | 0.45 (3) | 0.45 (2) | 0.50 (2) |
| general purpose | unpaced | 1.20 (1) | 0.68 (78) | 1.30 (9) | 0.43 (3) | 0.67 (2) | 0.93 (2) |
- Once a writer is slower than the network, a flush is 3–10× shorter than
writing everything at the flush, and the whole run is shorter with it.
- Small objects give the shortest flush, at the cost of one object per few MiB:
more requests, and reads coalesce less.
- Large objects keep a checkpoint to a few objects. Their flush also waits for the
open object's last piece and its completion: 0.4–0.5 s on a general purpose
bucket, 0.2 s on a directory bucket.
- A writer faster than the network gains little: the data can leave no faster
than the network takes it.
- The longest single write is a few milliseconds for objects sent whole, and up to
a few hundred for objects filled in pieces when the writer outruns the network
and waits for room.
## Write amplification
A block sent ahead and written again is sent again: once more when the change
came before its object completed, and once more for every later rewrite either
way. A working set written over and over is the worst case. 512 MiB written at
200 MB/s round a working set, data uploaded against the data kept:
| 512 MiB (no rewrite) | 1.0× | 1.0× | 1.0× | 1.0× |
| 64 MiB | 1.0× | 8.0× | 8.0× | 2.0× |
| 16 MiB | 1.0× | 26–32× | 9–17× | 2.0× |
With small objects each completes before the set comes round again, so every
round is sent in full. With large objects a block already in the open object
is not sent again: its first copy dies when it is written again, and the flush
writes its last bytes — twice in all. The counters are in `FlushTiming`:
`stream_sent_bytes`, `stream_dead_bytes` (died before its object counted),
`stream_flush_bytes` (written by the flush itself) and `stream_meta_bytes`.
`tests/bench_stream_flush.rs` reproduces this, with `BENCH_HOT_MIB` for the
working set.