Skip to main content

Module text

Module text 

Source
Expand description

UTF-8-safe text truncation for context render and compression.

All byte-index cuts must land on char boundaries — slicing mid-scalar panics in debug builds and produces invalid strings in release.

Functions§

proportional_byte_keep
Proportional byte budget for render-time truncation: keep remaining of total estimated tokens from a message whose content is text bytes long.
truncate_bytes_at_char_boundary
Return the longest prefix of text with at most max_bytes UTF-8 bytes, never splitting a scalar value.
truncate_with_suffix
Truncate text to at most max_bytes bytes on a char boundary and append suffix.