Expand description
QUITO-X–style trade-off: compress by dropping low token-entropy lines while targeting an output/input token ratio.
Query-conditioned variant (#542, EFF-5): the IB objective is
min I(T;X) − β·I(T;Y) — the relevance variable Y (the task/query) must
condition the compression. compress_ib_with_query fuses normalized
entropy with an IDF-weighted query-term overlap (the lexical core of
BM25), so two different queries keep different lines from the same file
(QUITO-X EMNLP’25: query-conditioned beats query-agnostic by 20-25%
accuracy at equal rate). Without a query the behavior is byte-identical
to the entropy-only path.
Functions§
- compress_
ib - Compress
texttowardtarget_ratio(output tokens / input tokens) by dropping lines whose normalized BPE token entropy falls below a dynamically chosen threshold. - compress_
ib_ with_ query - Query-conditioned IB compression (#542). With a query, the keep-score is
0.5·entropy + 0.5·relevance; without one (or when the query shares no terms with the document) this is exactly the entropy-onlycompress_ib.