Module dictionary

Available on crate feature dict-builder only.

Expand description

Dictionary training.

Effective dictionaries are up to 1% the size of the complete training body, and are trained on many examples of the original data. The trainers are those of the reference implementation, each taking the samples back to back with their sizes:

COVER (train_cover_dict, optimize_cover_dict) indexes every dmer exactly and fills the dictionary with the segments whose dmers the most samples share.
FastCOVER (train_fastcover_dict, optimize_fastcover_dict) does the same over a hashed frequency table, faster and with less memory; it is what zstd --train runs.
The legacy trainer (create_legacy_dict_from_slice) searches a suffix array for repeated segments.

The optimize_* forms search segment and dmer sizes, scoring each candidate by the total size of the scoring samples compressed with it.

create_raw_dict_from_slice and its reader and directory forms build raw content, with no entropy tables, from a corpus without sample sizes: the FastCOVER search zstd --train runs, over the corpus cut into samples.

Structs§

CoverOptions: Tuning for COVER training, the knobs of the reference’s ZDICT_cover_params_t.
FastCoverOptions: Tuning for FastCOVER training, the knobs of the reference’s ZDICT_fastCover_params_t.
FinalizeOptions: How a dictionary is finalized and trained against, the reference’s ZDICT_params_t.

Enums§

TrainingError: Why a segment trainer or the finalizer refused its input. It rides inside the InvalidInput error returned; TrainingError::of reads it back.

Constants§

DEFAULT_SELECTIVITY: Selectivity the reference uses when none is given (g_selectivity_default).
MIN_TRAINED_DICT_SIZE: Smallest dict_size a dictionary is trained or finalized into (upstream zstd ZDICT_DICTSIZE_MIN).

Functions§

check_cover_options: Refuse COVER tuning no training run can use for a dict_size dictionary, from the options alone: what train_cover_dict (both k and d given) or optimize_cover_dict (either left zero) would refuse before reading a sample. Lets a caller holding a large corpus fail before loading it.
check_cover_sample_count: Refuse sample_count samples the COVER training options select cannot build from: too few once the split takes its scoring share, or none left to score on. Needs only the count, so a caller can ask before reading a sample.
check_fastcover_options: check_cover_options for FastCOVER: also refuses f, accel and a d below 4 as train_fastcover_dict and optimize_fastcover_dict do.
check_fastcover_sample_count: check_cover_sample_count for FastCOVER, whose search scores on a quarter of the samples unless split_point says otherwise.
check_finalize_dict_size: Refuse a dict_size finalize_raw_dict cannot fill, from the size alone: below MIN_TRAINED_DICT_SIZE. Finalizing checks it first, before the samples; a caller holding the samples somewhere costly to walk can run it before that.
create_legacy_dict_from_slice: Train and finalize a dictionary with the reference’s original trainer, the one zstd --train-legacy runs (ZDICT_trainFromBuffer_legacy).
create_raw_dict_from_dir: Create a “raw content” dictionary of at most dict_size bytes, with no entropy tables, from every file in this directory and its subdirectories, and write it to output.
create_raw_dict_from_slice: Create a “raw content” dictionary of at most dict_size bytes, with no entropy tables, from a corpus already in memory, writing it to output.
create_raw_dict_from_source: Read from source to create a “raw content” dictionary of dict_size. The completed dictionary is written to output.
finalize_raw_dict: Finalize raw dictionary content into a dictionary of at most dict_size bytes: magic, id, entropy tables, repeat offsets and the content (the reference’s ZDICT_finalizeDictionary).
optimize_cover_dict: Train COVER dictionaries over a range of k and d and keep the one the scoring samples compress best with (the reference’s ZDICT_optimizeTrainFromBuffer_cover). Returns it with the options that built it.
optimize_fastcover_dict: Train FastCOVER dictionaries over a range of k and d and keep the one the scoring samples compress best with (the reference’s ZDICT_optimizeTrainFromBuffer_fastCover, which zstd --train and ZDICT_trainFromBuffer run). Returns it with the options that built it.
train_cover_dict: Train a COVER dictionary of at most dict_size bytes with the k and d given (the reference’s ZDICT_trainFromBuffer_cover).
train_fastcover_dict: Train a FastCOVER dictionary of at most dict_size bytes with the k and d given (the reference’s ZDICT_trainFromBuffer_fastCover).