combiner 0.1.2

A tool for recursively combining text files in a directory
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89

# Combiner

Combiner is a Rust-based command-line tool that processes text files in a given directory, combining their contents into a single output file. This tool is particularly useful for providing context to Large Language Models (LLMs) about the files in a project, streamlining the process of getting debugging advice or a project overview.

## Features

- Recursively scans directories for text files
- Configurable file inclusion/exclusion patterns
- Parallel processing for improved performance
- Token counting using the tiktoken-rs library
- Detailed output statistics
- Support for both command-line arguments and configuration files

## Installation

### Prerequisites

- Rust programming language (<https://www.rust-lang.org/tools/install>)

### Building from source

1. Clone the repository:

   ```
   git clone https://github.com/yourusername/combiner.git
   cd combiner
   ```

2. Build the project:

   ```
   cargo build --release
   ```

3. The binary will be available at `target/release/combiner`

Alternatively, you can use install combiner using cargo:

```
cargo install combiner
```

This will build the project and copy the binary to `/usr/local/bin/combiner`.

## Usage

Basic usage:

```
combiner -d <input_directory> -o <output_file>
```

For more options:

```
combiner --help
```

### Command-line Options

- `-d, --input-dir <input_dir>`: Input directory to process (default: current directory)
- `-o, --output-file <output_file>`: Output file path
- `-g, --ignore-patterns <ignore_patterns>`: Patterns to ignore (in addition to those in config)
- `-c, --config-file <config_file>`: Path to config file
- `-v, --verbose`: Enable verbose output

### Configuration File

You can use a TOML configuration file to set default options. By default, the program looks for a `combiner.toml` file in the input directory. You can specify a different config file using the `-c` option.

Example `combiner.toml`:

```toml
ignore_patterns = ["*.log", "*.tmp"]
include_patterns = ["*.rs", "*.toml"]
output_file = "combined_output.txt"
```

## Output

The program generates a single output file containing the contents of all processed text files. Each file's content is preceded by its file path and separated by a line of dashes.

The program also prints a summary table showing:

- Number of files processed
- Total number of tokens
- Output file path
- Processing time
- Top files by token count