Expand description
Rows per chunk on the background write paths (§5.1.5, D-011, D-014, D-058).
The Write Actor holds the sole write connection, so a single large statement
blocks every other writer for its duration. Chunking bounds that stall; the
cost is that a bulk import is not atomic across chunks, which is why it is
a separate command from HighPriCommand::WriteBulkAtomic rather than a
tuning parameter on it.
§Why these are four constants and not one
Through 0.5.5 this was a single CHUNK_ROWS = 1000 for all four bulk paths.
The golden rule it was meant to serve is a bound on duration — a background
chunk must commit fast enough that an interactive write queued behind it is
not made to wait — and one row count cannot express one duration across paths
whose measured per-row costs differ by 60× (D-058). At 1,000 rows the four
paths took 3.5 ms, 24 ms, 89 ms and 143 ms: the same constant, four answers,
three of them far outside the bound.
Each size below is derived from benches/budgets.rs’s chunk_scaling
sweep against CHUNK_BUDGET, then verified by measuring that size directly.
They are measurements of this machine, not universal constants — D-055’s
reasoning about reference hardware applies here too, and re-deriving them on
materially different storage is a cargo bench away.
§Sized for the tail, not the median
The first derivation solved f + c·n = 3 ms exactly and produced sizes whose
median commit was 2.93 ms and whose upper estimate was 2.96 — inside the
bound as reported and outside it for any chunk slower than typical. A latency
bound is a statement about the chunk an unlucky interactive write actually
queues behind, so these solve for ≈2.5 ms instead, leaving the remainder as
headroom for the tail. That costs a few percent of throughput on the two
linear paths and nothing on the two superlinear ones.
As measured by chunk_budget, each at its own size: edges 2.39 ms,
concepts 2.35 ms, annotations 2.36 ms, embeddings 2.06 ms, no
upper estimate above 2.42.
§Known limitation: these are empty-database figures
chunk_budget seeds concepts and starts with no links and no vectors,
and D-059 established that per-row cost on the edge and embedding paths grows
with the size of the structure being written, not with the chunk. The same
90-edge chunk takes 47.7 ms against an 8,000-edge hub. So the bound is met as
measured and not met on a large database, most of that gap being the schema
defect D-059 documents. Re-deriving these against a realistic fixture needs a
decision about what “realistic” is, which is why it has not been done
silently.
Constants§
- ANNOTATIONS
- Analytics annotations (
write_analytics_annotations). - CONCEPTS
- Concept upserts (
write_concepts). - EDGES
- Edge assertions (
bulk_import). - EMBEDDINGS
- Embedding vectors (
upsert_embeddings).