Modules§
Functions§
- concat_
files - count_
tokens - Count tokens using tiktoken’s GPT-3.5-Turbo tokenizer for accuracy
- is_
text_ file - Check if a file is likely text or binary by reading only a small chunk. This avoids reading large files fully just to detect their type.
- parse_
token_ limit - Parse a token limit string like “800k” or “1000” into a number
- serialize_
repo - Main entrypoint for serialization, used by CLI and tests