Crate yek

Crate yek 

Source

Modules§

config
defaults
parallel
priority
tree

Functions§

concat_files
count_tokens
Count tokens using tiktoken’s GPT-3.5-Turbo tokenizer for accuracy
is_text_file
Check if a file is likely text or binary by reading only a small chunk. This avoids reading large files fully just to detect their type.
parse_token_limit
Parse a token limit string like “800k” or “1000” into a number
serialize_repo
Main entrypoint for serialization, used by CLI and tests