# Stream segments: writing a checkpoint while it is produced
`BlockPtrFormat::StreamSegment` lays every object out tail first, so data can go
to storage while it is still being written instead of all at once when a flush
asks for it. A flush is left with what has not been sent yet, plus a small part 0.
The format is chosen when a container is created, like any block-pointer format.
How data is sent ahead is a runtime setting, `stream_segment_bytes`, and can differ
between sessions on the same container. Builds with and without the WAL write the
same layout, so a container can be written by either in turn; see
[With the WAL](#with-the-wal).
## Layout
```text
[data blocks][meta blocks][summary entries][header]
n × data m × meta n × 16 256 B
```
A data block's place is `k × data block size` the moment it is placed, whatever
comes after it, so its pointer is final before the object is finished. The index
nodes point at the data, so they come after it. The summary lists the data blocks.
The header comes last, because it counts everything, names the object and carries
the inode.
The header is self-contained: magic, layout version, flags, checkpoint number and
part, block sizes, block counts, the inode the checkpoint was built on, the inode,
and a CRC-32. A reader finds it in an object's last 256 bytes and refuses it unless
the magic, version and CRC hold, the counts add up to exactly the object's length,
and it names the object it was read from. A layout version newer than the build is
`Unsupported`.
Objects are named as under `PartedSegment`: `<cno>.<part>`, part 0 being the
checkpoint. A reader that knows the format reads an object's end with a suffix
request, which usually returns the header, the summary and the index nodes
together. A reader that does not know the format (opening a given checkpoint, the
cleaner) looks at the end of a part first and reads it from its start when the
end is not a stream header. An unsuffixed object is never tail first.
## Every object describes itself
Each object holds its data, the leaves of the block map over that data, a summary
naming every block, and a header. A leaf is never split from its data: the blocks
under one leaf go into one object, and that leaf goes with them. An object can
therefore be a little over its target. Part 0 holds the index nodes above the
leaves, the leaves with no new data under them, whatever data still fits, and the
inode.
A block written again after it was placed in an object leaves a dead slot there.
Its summary entry says so, the block stays dirty, and a later object or the flush
writes it again.
## Sending ahead: `stream_segment_bytes`
The target size of the objects data is sent in while it is written. It is capped
at `segment_max_object_bytes`. Default 1 GiB.
| `0` | nothing sent ahead | as many as the flush needs | all the data |
| ≤ one `segment_mpu_chunk_size` piece | each object sent whole, in one request, once there is one's worth | many small ones | the last small object |
| larger | one object filled a piece at a time and closed once it holds the target | few, about the target | the open object's last piece and its completion |
In both ways of sending ahead, at most `segment_mpu_max_concurrency` objects or
pieces are in flight, each holding a copy of its blocks. A block leaves the dirty
tier once the object holding it is complete, if two things still hold: its
pointer is where it was sent, and a hash of its bytes matches what was sent. A
leaf whose bytes are unchanged and that has no dirty block under it becomes clean.
Nothing in a multipart upload is readable until it completes. So with objects
filled in pieces, the blocks of the open object stay in memory until it closes,
and the dirty-data limits still apply.
At a flush, whatever is still unsent goes up as pieces, several at once, and the
open object closes. Part 0 is written after every object has completed, then the
inode. The checkpoint's number is taken once. Every object and every attempt of
the flush uses it, and each attempt reserves its part numbers before writing.
An object that cannot complete is aborted. Its blocks are still dirty, so the
flush writes them and nothing is lost. A write whose answer was lost and which the
SDK resent completes, as any conditional write does; see `x-amz-meta-hf-write` in
[concurrency](concurrency.md).
## The dirty-data limits come first
**With the default limits nothing is sent ahead.** An object can be no larger
than the dirty data the file may hold — its blocks stay in memory until it is
complete — and that is
```text
min(max(data_cache_dirty_max_bytes_threshold,
data_cache_dirty_max_blocks_threshold × data_block_size),
segment_buffer_size)
```
which is 8 MiB by default: less than one `segment_mpu_chunk_size` piece, 16 MiB.
The limit is crossed before a piece's worth is unsent, and crossing it writes a
partial segment the writer waits for, as under `PartedSegment`. Measured, 512 MiB
in 1 MiB writes with the default limits:
| directory | 5.2 s, 57 objects of ~9 MiB | 5.1 s, the same objects, nothing sent ahead | 5.1 s, the same |
| general purpose | 8.0 s | 8.1 s | 8.0 s |
against 1.2–3.4 s with the limits raised. The longest single write was 0.1–0.2 s,
waiting for a partial.
So raise the limits with the format, to at least the object size wanted plus the
pieces in flight:
```rust
rt.data_cache_dirty_max_bytes_threshold = 1 << 30; // and memory to match
rt.segment_buffer_size = 1 << 30;
rt.stream_segment_bytes = 1 << 30; // the default
```
With objects sent whole (`stream_segment_bytes` up to one piece), the limits need
only hold a few objects — `segment_mpu_max_concurrency` of them in flight, each
also copied while it uploads. With objects filled in pieces they must hold a whole
object: one closes early, at whatever size it has, when the limits are crossed.
## With the WAL
A build with the WAL writes checkpoints in the same layout and reads those any
build wrote. Three paths publish, and each writes the stream layout:
- the reactor's WAL publish builds the checkpoint as one object, part 0, keeps it
pinned in memory while it uploads, and serves reads from it until it lands, as
it does under the other formats; with the WAL a checkpoint published this way is
always one object, and one larger than `segment_max_object_bytes` is refused;
- a flush on the direct API, and a replay at open, lay the checkpoint out as a
build without the WAL does, in parts when it is larger than one object;
- memory pressure inside a transaction writes partials, as elsewhere.
Nothing is sent ahead in a build with the WAL: `stream_segment_bytes` is not used
there. The reactor's WAL publish already hands the upload to the background and
lets reads and writes go on.
## What it buys
512 MiB written in 1 MiB writes at a fixed rate, then one flush.
`segment_max_object_bytes` 1 GiB, pieces 16 MiB, and the dirty-data limits raised
out of the way (see above). The reactor's handler (`fh_write`, `fh_flush`) gives
the same figures as the direct API. Flush time in seconds, with
objects per checkpoint in parentheses:
| directory | 100 MB/s | 0.74 (1) | 0.07 (129) | 0.18 (33) | 0.17 (3) | 0.18 (2) | 0.19 (2) |
| directory | 400 MB/s | 0.74 (1) | 0.08 (128) | 0.17 (33) | 0.17 (3) | 0.18 (2) | 0.19 (2) |
| general purpose | 100 MB/s | 1.22 (1) | 0.17 (129) | 0.42 (33) | 0.45 (3) | 0.43 (2) | 0.48 (2) |
| general purpose | 400 MB/s | 1.18 (1) | 0.58 (108) | 0.76 (29) | 0.45 (3) | 0.45 (2) | 0.50 (2) |
| general purpose | unpaced | 1.20 (1) | 0.68 (78) | 1.30 (9) | 0.43 (3) | 0.67 (2) | 0.93 (2) |
- Once a writer is slower than the network, a flush is 3–10× shorter than
writing everything at the flush, and the whole run is shorter with it.
- Small objects give the shortest flush, at the cost of one object per few MiB:
more requests, and reads coalesce less.
- Large objects keep a checkpoint to a few objects. Their flush also waits for the
open object's last piece and its completion: 0.4–0.5 s on a general purpose
bucket, 0.2 s on a directory bucket.
- A writer faster than the network gains little: the data can leave no faster
than the network takes it.
- The longest single write is a few milliseconds for objects sent whole, and up to
a few hundred for objects filled in pieces when the writer outruns the network
and waits for room.
## Write amplification
A block sent ahead and written again is sent again: once more when the change
came before its object completed, and once more for every later rewrite either
way. A working set written over and over is the worst case. 512 MiB written at
200 MB/s round a working set, data uploaded against the data kept:
| 512 MiB (no rewrite) | 1.0× | 1.0× | 1.0× | 1.0× |
| 64 MiB | 1.0× | 8.0× | 8.0× | 2.0× |
| 16 MiB | 1.0× | 26–32× | 9–17× | 2.0× |
With small objects each completes before the set comes round again, so every
round is sent in full. With large objects a block already in the open object
is not sent again: its first copy dies when it is written again, and the flush
writes its last bytes — twice in all. The counters are in `FlushTiming`:
`stream_sent_bytes`, `stream_dead_bytes` (died before its object counted),
`stream_flush_bytes` (written by the flush itself) and `stream_meta_bytes`.
`tests/bench_stream_flush.rs` reproduces this, with `BENCH_HOT_MIB` for the
working set.