Module information_bottleneck 

QUITO-X–style trade-off: compress by dropping low token-entropy lines while targeting an output/input token ratio.

Functions

compress_ib

Compress text toward target_ratio (output tokens / input tokens) by dropping lines whose normalized BPE token entropy falls below a dynamically chosen threshold.