Module encoding 

File encoding handling (Phase 10)

This module provides robust handling of file encodings during analysis.

Mitigations

• A34: Silent data corruption on non-UTF8 files
  • Detects and handles UTF-8 BOM
  • Falls back to lossy decoding with warning
  • Detects and skips binary files

Example

use tldr_core::encoding::{read_source_file, FileReadResult};
use std::path::Path;

match read_source_file(Path::new("example.py")) {
    Ok(FileReadResult::Ok(content)) => {
        // Process UTF-8 content
    }
    Ok(FileReadResult::Lossy { content, warning }) => {
        // Process with warning
        eprintln!("Warning: {}", warning);
    }
    Ok(FileReadResult::Binary) => {
        // Skip binary file
    }
    Err(e) => {
        // Handle IO error
    }
}

Structs

EncodingIssue

A single encoding issue record.

EncodingIssues

Record of encoding issues encountered during analysis.

Enums

FileReadResult

Result of reading a source file with encoding detection.

Functions

is_binary_file

Check if a file appears to be binary.

read_source_file

Read a source file with encoding detection.

read_source_file_or_skip

Read a source file, returning the content or skipping on error.