aptitude 0.3.0

A test harness for validating AI agent behavior against steering guides
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143

# Aptitude

A test harness for validating AI agent (Claude Code) behavior against steering guides. This tool executes Claude with prompts and asserts on the tool calls made.

## Installation

### From crates.io

```bash
cargo install aptitude
```

### From Homebrew

```bash
brew tap tatimblin/aptitude
brew install aptitude
```

### From source

```bash
git clone https://github.com/tatimblin/aptitude
cd aptitude
cargo build --release
```

## Usage

The harness has two main commands:

### Run tests

Execute Claude with a test file and evaluate assertions:

```bash
# Run a single test
aptitude run test.yaml

# Run all tests in a directory
aptitude run tests/

# With verbose output and custom working directory
aptitude run test.yaml -v -w /path/to/workdir
```

### Analyze existing sessions

Analyze a pre-existing Claude session log against test assertions:

```bash
aptitude analyze test.yaml session.jsonl
```

## Test File Format

Tests are defined in YAML files:

```yaml
name: "Test name"
prompt: "The prompt to send to Claude"
assertions:
  - tool: Read
    called: true
    params:
      file_path: "*.txt"  # glob pattern
  - tool: Bash
    called: false
  - tool: Write
    called_after: Read
```

### Assertion Types

- `called: true/false` - Whether a tool was called
- `params` - Parameter matching (supports glob patterns, regex, or exact match)
- `called_after` - Ordering assertions (tool A must be called after tool B)

## Development

```bash
# Build the project
cargo build

# Run tests
cargo test

# Run a specific test
cargo test test_name
```

## Release Process

This project has fully automated releases to both crates.io and Homebrew:

### One-time setup

1. **Create crates.io API token:**
   - Go to https://crates.io/settings/tokens and create a token
   - Add it as `CARGO_REGISTRY_TOKEN` secret in GitHub repo settings

2. **Create GitHub personal access token:**
   - Go to GitHub Settings → Developer settings → Personal access tokens
   - Create a token with `repo` permissions for your homebrew-tap repo
   - Add it as `HOMEBREW_TAP_TOKEN` secret in GitHub repo settings

3. **Create Homebrew tap repository:**
   ```bash
   gh repo create tatimblin/homebrew-aptitude --public
   ```

### Create a new release

```bash
# Bump version and trigger fully automated release
./release.sh patch  # 0.1.0 -> 0.1.1
./release.sh minor  # 0.1.0 -> 0.2.0
./release.sh major  # 0.1.0 -> 1.0.0
./release.sh 1.2.3  # specific version
```

This single command automatically:
1. Updates version in `Cargo.toml` and `Cargo.lock`
2. Creates and pushes a git tag
3. Triggers GitHub Actions that:
   - Builds binaries for all platforms
   - Creates GitHub release with binaries
   - Publishes to crates.io
   - Calculates SHA256 hashes for Homebrew formula
   - Updates and pushes the Homebrew formula to your tap

**No manual steps required!**

## Architecture

- **parser.rs** - Parses Claude Code JSONL session logs
- **assertions.rs** - Test structure and assertion evaluation
- **executor.rs** - Executes Claude and finds session logs
- **watcher.rs** - File watching for incremental parsing

## License

MIT