Constants§
- TRUNCATION_
MARKER - The elision marker appended when
truncate_to_tokensdrops content.
Functions§
- compress_
block - Collapse repeated blank lines while preserving paragraph breaks, and compress whitespace within each line.
- compress_
text - Semantic text reduction: strip decorative glyphs, then collapse runs of whitespace, then trim.
- estimate_
tokens - Fast token approximation.
- truncate_
to_ tokens - Truncate text to roughly
max_tokens, on a character boundary, appending an elision marker when content is dropped. - truncate_
to_ tokens_ at truncate_to_tokens, also reporting how many bytes oftextthe returned prefix consumed.- window_
from - The remainder of
textfrom a byteoffset, for reading a long document a window at a time.