Skip to main content

Module compress

Module compress 

Source

Constants§

TRUNCATION_MARKER
The elision marker appended when truncate_to_tokens drops content.

Functions§

compress_block
Collapse repeated blank lines while preserving paragraph breaks, and compress whitespace within each line.
compress_text
Semantic text reduction: strip decorative glyphs, then collapse runs of whitespace, then trim.
estimate_tokens
Fast token approximation.
truncate_to_tokens
Truncate text to roughly max_tokens, on a character boundary, appending an elision marker when content is dropped.
truncate_to_tokens_at
truncate_to_tokens, also reporting how many bytes of text the returned prefix consumed.
window_from
The remainder of text from a byte offset, for reading a long document a window at a time.