Skip to main content

Module write_opts

Module write_opts 

Source
Expand description

Writer tuning knobs: index tile size, dictionary chunk size, zstd level and frame shape, and frequency-promoted object ids. The measurements behind the defaults are in docs/BENCHMARK.md (“Compression (0.4)”).

None of these change the file format. Every combination writes a file any reader of the same format version opens unchanged: tiles and dictionary chunks of any size are routed by their directories, zstd frames of any level or shape decode with any conformant decoder, and a promoted object is an ordinary entry of the shared dictionary section.

The settings are process-wide, like RETE_DICT_RESTART_INTERVAL: the CLI sets them once, before a build, and every writer path (the in-memory and streaming assemblers and the external build) reads the same values, so the byte-identity between those paths holds for every setting. Each knob falls back to an environment variable, then to the built-in default:

knobenv vardefault
index tile budgetRETE_TILE_KIB64 KiB
dictionary chunk budgetRETE_DICT_CHUNK_KIB64 KiB
zstd levelRETE_ZSTD_LEVEL9
zstd frame shapeRETE_ZSTD_FRAME (sized / stream)sized
frequency-promoted objectsRETE_FREQUENT_OBJECTS0 (off)

Structs§

WriteOptions
A complete set of writer settings.

Enums§

FrameShape
How each tile / chunk is framed by zstd.

Constants§

DEFAULT_DICT_CHUNK_BYTES
Default per-chunk budget of a dictionary section body, in uncompressed bytes.
DEFAULT_TILE_BUDGET_BYTES
Default per-tile budget of an index permutation, in uncompressed bytes.
DEFAULT_ZSTD_LEVEL
Default zstd level of the writer.
MAX_BUDGET_KIB
Largest accepted tile or chunk budget (KiB). A tile is decoded whole on every touch, so very large tiles turn a point lookup into a large decode.
MIN_BUDGET_KIB
Smallest accepted tile or chunk budget (KiB). Below this the per-unit directory entry and frame header dominate.

Functions§

dict_chunk_budget
The dictionary chunk budget in force (bytes).
frame_shape
The zstd frame shape in force.
frequent_objects
How many frequent object-only terms the dictionary builder promotes.
set_write_options
Replace the settings in force for this process (the CLI calls this once, from its flags, before a build). Values outside the accepted ranges are clamped.
tile_budget
The index tile budget in force (bytes).
write_options
The full set of settings in force.
zstd_level
The zstd level in force.