# Combiner
Combiner is a Rust-based command-line tool that processes text files in a given directory, combining their contents into a single output file. This tool is particularly useful for providing context to Large Language Models (LLMs) about the files in a project, streamlining the process of getting debugging advice or a project overview.
## Features
- Recursively scans directories for text files
- Token counting using the tiktoken-rs library
- Rayon-based parallel processing for faster processing
- Detailed output statistics
## Installation
### Prerequisites
- Rust programming language (<https://www.rust-lang.org/tools/install>)
### Building from source
1. Clone the repository:
```
git clone https://github.com/jesalx/combiner.git
cd combiner
```
2. Build the project:
```
cargo build --release
```
3. The binary will be available at `target/release/combiner`
Alternatively, you can use install combiner using cargo:
```
cargo install combiner
```
## Usage
Basic usage:
```
combiner -d <directory> -o <output> -t <tokenizer>
```
For more options:
```
combiner --help
```
### Command-line Options
- `-d, --directory <directory>`: Input directory to process (default: current directory)
- `-o, --output <output>`: Output file path/name
- `-t, --tokenizer <tokenizer>`: Tokenizer to use (default: p50k_base)
## Output
The program generates a single output file containing the contents of all processed text files. Each file's content is preceded by its file path and separated by a line of dashes.
The program also prints a summary table showing:
- Number of files processed
- Total number of tokens
- Output file path
- Processing time
- Top file by token count