Expand description
Writer tuning knobs: index tile size, dictionary chunk size, zstd level and
frame shape, and frequency-promoted object ids. The measurements behind the
defaults are in docs/BENCHMARK.md (“Compression (0.4)”).
None of these change the file format. Every combination writes a file any reader of the same format version opens unchanged: tiles and dictionary chunks of any size are routed by their directories, zstd frames of any level or shape decode with any conformant decoder, and a promoted object is an ordinary entry of the shared dictionary section.
The settings are process-wide, like RETE_DICT_RESTART_INTERVAL: the
CLI sets them once, before a build, and every writer path (the in-memory and
streaming assemblers and the external build) reads the same values, so the
byte-identity between those paths holds for every setting. Each knob falls
back to an environment variable, then to the built-in default:
| knob | env var | default |
|---|---|---|
| index tile budget | RETE_TILE_KIB | 64 KiB |
| dictionary chunk budget | RETE_DICT_CHUNK_KIB | 64 KiB |
| zstd level | RETE_ZSTD_LEVEL | 9 |
| zstd frame shape | RETE_ZSTD_FRAME (sized / stream) | sized |
| frequency-promoted objects | RETE_FREQUENT_OBJECTS | 0 (off) |
Structs§
- Write
Options - A complete set of writer settings.
Enums§
- Frame
Shape - How each tile / chunk is framed by zstd.
Constants§
- DEFAULT_
DICT_ CHUNK_ BYTES - Default per-chunk budget of a dictionary section body, in uncompressed bytes.
- DEFAULT_
TILE_ BUDGET_ BYTES - Default per-tile budget of an index permutation, in uncompressed bytes.
- DEFAULT_
ZSTD_ LEVEL - Default zstd level of the writer.
- MAX_
BUDGET_ KIB - Largest accepted tile or chunk budget (KiB). A tile is decoded whole on every touch, so very large tiles turn a point lookup into a large decode.
- MIN_
BUDGET_ KIB - Smallest accepted tile or chunk budget (KiB). Below this the per-unit directory entry and frame header dominate.
Functions§
- dict_
chunk_ budget - The dictionary chunk budget in force (bytes).
- frame_
shape - The zstd frame shape in force.
- frequent_
objects - How many frequent object-only terms the dictionary builder promotes.
- set_
write_ options - Replace the settings in force for this process (the CLI calls this once, from its flags, before a build). Values outside the accepted ranges are clamped.
- tile_
budget - The index tile budget in force (bytes).
- write_
options - The full set of settings in force.
- zstd_
level - The zstd level in force.