context-creator 1.5.0

High-performance CLI tool to convert codebases to Markdown for LLM context
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Overview

context-creator is a high-performance Rust CLI tool that converts entire codebases into LLM-optimized Markdown for consumption by models like Gemini. It processes git repositories by intelligently filtering, prioritizing, and formatting code into a single cohesive document.

## Key Commands

### Development Commands
```bash
# Build and run validations (format check + lint)
make build

# Run tests with validation
make test

# Run in development mode with example
make dev

# Install locally
make install

# Generate documentation
make doc
```

### Quality & CI Commands
```bash
# Format code
make fmt

# Check formatting (CI)
make fmt-check

# Lint with clippy
make lint

# Run all validation checks
make validate

# Generate test coverage
make coverage
```

### Testing Commands
```bash
# Run all tests
cargo test

# Run specific test
cargo test test_name

# Run tests with specific features
cargo test --all-targets

# Run benchmarks
cargo bench
```

## Project Architecture

### Core Components

1. **CLI Layer (`src/cli.rs`)**
   - Command-line argument parsing using clap
   - Configuration validation and loading
   - Supports multiple input sources: directories, glob patterns, GitHub repos

2. **Core Processing (`src/core/`)**
   - `walker.rs`: Directory traversal with .gitignore support and custom filtering
   - `context_builder.rs`: Markdown generation with token management
   - `prioritizer.rs`: File prioritization based on importance and token limits
   - `semantic/`: Language-aware semantic analysis for imports and dependencies
   - `cache.rs`: File caching for performance optimization

3. **Configuration System (`src/config.rs`)**
   - TOML-based configuration with `.context-creator.toml`
   - Support for custom priorities, token limits, and ignore patterns
   - Hierarchical configuration loading (CLI > config file > defaults)

4. **Semantic Analysis (`src/core/semantic/`)**
   - Multi-language AST parsing using tree-sitter
   - Import tracing and dependency resolution
   - Language-specific analyzers in `languages/` directory
   - Supports 20+ programming languages

### Key Data Flow

1. **Input Processing**: CLI args → Config validation → Directory resolution
2. **File Discovery**: Walker scans directories → Applies ignore patterns → Filters by include patterns
3. **Semantic Analysis**: Optional import tracing → Dependency resolution → Enhanced file relationships
4. **Prioritization**: File importance scoring → Token budget allocation → Selection for output
5. **Output Generation**: Markdown formatting → Token counting → Final document assembly

### Configuration Files

- `.contextignore`: Exclude files/directories (gitignore syntax)
- `.contextkeep`: Prioritize important files
- `.context-creator.toml`: Advanced configuration (priorities, token limits, defaults)

### Token Management

The tool implements sophisticated token budget management:
- Automatic token counting using tiktoken-rs
- Prompt token reservation for LLM interactions
- Intelligent file truncation and selection
- Per-LLM token limit configuration

### Semantic Features

- **Import Tracing**: Follow import chains across files
- **Caller Analysis**: Find functions that call specific modules
- **Type Dependencies**: Include type definitions and interfaces
- **Multi-language Support**: Unified analysis across different languages

## Important Implementation Details

### Error Handling
- Uses `anyhow` for error propagation
- Custom error types in `src/utils/error.rs`
- Comprehensive validation at multiple levels

### Performance Optimizations
- Parallel processing with rayon
- File caching to avoid re-reading
- Tree-sitter parser pooling
- LRU caching for AST parsing

### Testing Strategy
- Unit tests for core functionality
- Integration tests for CLI combinations
- Performance benchmarks
- Semantic analysis stress tests
- Security vulnerability tests

## Development Notes

### Adding New Languages
Language support is added in `src/core/semantic/languages/`:
1. Implement language-specific analyzer
2. Add tree-sitter grammar dependency
3. Update language registry in `mod.rs`

### Configuration Precedence
1. Explicit CLI arguments (highest priority)
2. Config file token limits (for prompts)
3. Config file defaults
4. Hard-coded defaults (lowest priority)

### File Prioritization
Base file type priorities can be customized via config file using glob patterns. Uses "first-match-wins" evaluation order.