Skip to main content

Module dictionary

limnifs_write

Module dictionary 

Source
Expand description

ZSTD dictionary training — writer-side API.

Wraps limnifs_core::codec::zstd_dict (which wraps omnizip_zstd::train_dictionary) at the writer layer. The codec-layer API takes raw samples; this layer adds:

  • Per-content-class sample collection (text, binary, etc.).
  • Configurable trainer (FrequencyTrainer today; FastCover via omnizip_zstd::FastCoverTrainer when needed).
  • Dictionary id allocation (0x00..=0xFE; 0xFF is NO_DICT).
  • Integration with WriteConfig::dictionaries.

§Pipeline integration (planned)

Today this module exposes the trainer; the writer pipeline does NOT yet call it. The plan, filed in TODO.impl/04-writer-pipeline/04-zstd-dictionary-training.md:

  1. Walk + parallel chunk + compress (existing pipeline).
  2. Collect unique plaintext drops per content class during merge_chunked_file.
  3. After parallel phase, train one dict per class with ≥ min_class_size drops.
  4. Re-compress eligible drops with compress_with_dict; keep the smaller of (original, dict-compressed).
  5. Drop records carry dict_id; manifest emits dictionary_section.

This module is the public API for steps 3–4. The pipeline glue lands in a follow-up PR.

Structs§

TrainedDictionary
Trained dictionary ready for use with compress_with_dict.

Enums§

TrainerKind
Trainer algorithm selection. See train_zstd_with_trainer.

Constants§

DEFAULT_MIN_SAMPLES
Minimum sample count before training is worthwhile. Below this, the trainer returns empty (not enough signal). Matches DictionaryConfig::min_class_size default.
DEFAULT_TARGET_SIZE
Default target dictionary size (64 KiB). Matches DictionaryConfig::max_dict_size default.

Functions§

allocate_ids
Allocate dictionary ids 0x00..=0xFE for a set of trained dicts. The 0xFF slot is reserved as NO_DICT sentinel.
train_zstd
Train a ZSTD dictionary from samples using the default FrequencyTrainer. Returns None if samples is empty, target size is 0, or the trainer produces an empty dictionary (not enough signal).
train_zstd_with_trainer
Train with explicit trainer selection. See train_zstd for the default-FrequencyTrainer shortcut.