Module entropy 

Structs

EntropyAnalysis

Per-line entropy statistics for a block of content.

EntropyResult

Result of entropy-based compression: output text, token counts, and techniques used.

Enums

CompressibilityClass

Classification of content compressibility based on Kolmogorov proxy (gzip ratio).

Functions

analyze_entropy

Analyzes per-line BPE token entropy, counting low/high entropy lines.

compressibility_class

Classify how compressible content is based on gzip ratio.

entropy_compress

Compresses content by removing low-entropy lines and deduplicating patterns.

entropy_compress_adaptive

Entropy compression with file-type-adaptive thresholds and event emission. force_keep lines (explicit protect tokens, #709) survive verbatim; pass &[] to keep the pre-protect behaviour byte-identical (#498).

entropy_compress_deterministic

Entropy compression with the opportunistic semantic redundancy filter (#544) pinned OFF. The regular path uses the shared embedding engine whenever it happens to be loaded, so its output depends on runtime state. Benchmarks and the scorecard (#211) need run-to-run and machine-to-machine reproducibility, so they must go through this entry point.

entropy_compress_task_conditioned

Task-conditioned entropy compression: lines that would normally be dropped for low entropy are kept if they contain task-relevant keywords. This is the Information Bottleneck proxy: we compress away only what is neither surprising (high H) nor task-relevant (mentions goal concepts). Falls back to pure entropy when task_keywords is empty.

entropy_compress_to_density

Budget-based compression to a target density (SDE principle: aim for a target information density instead of maximum compression).

entropy_compress_with_threshold

Like entropy_compress_adaptive but overrides the learned BPE-entropy threshold (e.g. from the aggressiveness knob) while keeping the file-adaptive jaccard. Pure function of its inputs (#498). Higher bpe_entropy drops more low-information lines.

jaccard_similarity

Computes word-set Jaccard similarity between two strings (0.0–1.0).

kolmogorov_proxy

Kolmogorov complexity proxy: K(x) ≈ len(gzip(x)) / len(x). Lower values = more compressible = more redundant.

minhash_signature

Minhash signature for approximate Jaccard via LSH. Uses k independent hash functions (polynomial hashing with different seeds).

minhash_similarity

Approximate Jaccard from two minhash signatures.

ngram_jaccard

N-gram Jaccard similarity — preserves word order (unlike word-set Jaccard).

normalized_token_entropy

Normalized Shannon entropy: H(X) / log₂(n) where n = number of unique symbols. Returns a value in [0, 1] where 0 = perfectly predictable, 1 = maximum entropy. This makes thresholds comparable across different alphabet sizes.

normalized_token_entropy_from_ids

Normalized Shannon entropy over encoded token IDs: H(X) / log₂(n), n = unique token count.

shannon_entropy

Computes Shannon entropy (bits) over character frequencies in the text.

token_entropy

Shannon entropy over BPE token IDs (o200k_base). More LLM-relevant than character entropy since LLMs process BPE tokens.

token_entropy_from_ids

Shannon entropy over already-encoded BPE token IDs (o200k_base).