Crate loctok

Structs§

aggregate_by_language
build_copy_output: Step 2: Build final output from relative paths and content collected in step 1. Format:
collect_filtered_texts: Step 1: Extract filtered relative file paths and their UTF-8 content. Returns (relative_path, content) for each file, sorted by path.
count_non_empty_lines
count_tokens_in_path
count_tokens_in_path_with_progress: Like count_tokens_in_path, but reports progress via the provided callback. The callback receives (processed_files, total_files).
count_tokens_in_text
get_encoder
language_from_path