Module text_decode 

Source-text decoding shared by every reader that feeds the indexes.

std::fs::read_to_string rejects anything that is not strict UTF-8, and the index builders used to continue past that error without a trace. On Windows that silently dropped whole projects: editors there still save “ANSI” (Windows-1252) files, and Visual Studio / PowerShell write UTF-16 with a byte-order mark. Such files were missing from BM25, the dense index, the graph and ctx_search, so ctx_compose answered “no match” for code that was plainly on disk.

decode never fails:

• a UTF-8 BOM is stripped (it is an encoding artifact, not content);
• a UTF-16 LE/BE BOM selects UTF-16;
• valid UTF-8 passes through without copying;
• otherwise the bytes are mostly UTF-8 with a few corrupt sequences (decoded lossily) or a legacy single-byte file (decoded as Windows-1252, a superset of Latin-1 that maps every byte, so line numbers stay exact).

Structs

Decoded

The one decoder outcome every decoding path produces.

Functions

looks_binary

NUL bytes mark binary data — except in UTF-16, where they are half the text.

read_text_checked

read_text with the decoder’s outcome: the text, the encoding it was decoded as, and whether any byte could not be represented. Readers that feed admission, caches or indexes use this form — a String alone cannot say that the text is not the whole source.