Expand description
Explicit, deterministic preservation of user-marked spans across every compressor (read, shell, proxy, prose).
Two complementary, fully deterministic mechanisms (#498 — pure functions of their inputs, no model, no clock, no global state):
- Universal markers
<lc_safe>…</lc_safe>: any compressor wraps its work incompress_preserving; the content between the markers passes through verbatim and the markers themselves are stripped from the output. protecttoken list (ctx_readconvenience):line_is_protectedlets the line-based lossy filters (entropy / information-bottleneck) force-keep every line that contains one of the given tokens.
Security: callers MUST run secret redaction before protect (redact → protect → compress). Protect never re-introduces redacted secrets because
it only ever passes through bytes that already survived redaction.
Constants§
- SAFE_
CLOSE - Closing marker for a verbatim-preserved span.
- SAFE_
OPEN - Opening marker for a verbatim-preserved span.
Functions§
- compress_
preserving - Compress only the unprotected regions of
inputwithf; everything between<lc_safe>and</lc_safe>passes through byte-for-byte and the markers are stripped from the output. - has_
markers - Cheap pre-check: does the input contain at least one protect marker? Lets hot paths skip the span-splitting machinery entirely when nothing is marked.
- line_
is_ protected - True if
linemust survive a lossy line filter because it contains one of the explicitprotecttokens. Empty tokens are ignored so an empty list (or a list of empty strings) reproduces today’s behaviour exactly. - protect_
fragment - Stable cache-key fragment for a
protecttoken list, or""when the list is empty (so unprotected reads keep their current cache key).