checkmate-cli 0.4.1

# Checkmate

**AI-native API testing framework with YAML specs and git-connected history.**

Checkmate makes API testing simple for both humans and AI agents. Write declarative test specs in YAML, run assertions with Clove queries, and track results with full git context.

## Quick Start

```bash
# Build from source
cargo build --release

# Initialize in your project
cm init --url http://localhost:8080

# Create a test spec
cat > .checkmate/tests/health.yaml << 'EOF'
name: "Health Check"
tests:
  api_responds:
    endpoint: /health
    assertions:
      - query: "$[doc][status]"
        expect: "ok"
EOF

# Run tests
cm test run
```

## Features

- **YAML Test Specs**: Declarative, readable test definitions
- **Clove Query Language**: Powerful JSON assertions with `$[field]` and `$[@prev]` scopes
- **API Version Diff**: Compare responses across endpoints to detect structural changes (`cm diff run`)
- **Variable Extraction**: Extract values from responses for multi-step auth flows
- **Universal Env Vars**: `${VAR}` expansion in endpoints, headers, bodies, and query params
- **Git-Connected History**: Every test run captures commit, branch, and dirty status
- **Auto-Discovery**: Run `cm test run` from any subdirectory
- **Configuration Hierarchy**: Defaults → User config → Project config → Environment
- **Hook System**: Run custom scripts on test lifecycle events
- **AI-Native**: Built-in documentation, Claude plugin, and agent-friendly CLI

## Installation

### From Source

```bash
git clone https://github.com/your-org/checkmate
cd checkmate
cargo build --release
# Binary at target/release/cm
```

### Add to PATH

```bash
export PATH="$PATH:/path/to/checkmate/target/release"
```

## Initialization

### Standard Mode

```bash
cm init --url http://localhost:8080
```

Creates `.checkmate/` directory structure:
```
.checkmate/
├── config.toml    # Project configuration
├── tests/         # Test specifications
├── runs/          # Test run history (JSONL)
└── hooks/         # Custom hook scripts
```

### Stealth Mode

For personal use on shared projects without committing Checkmate files:

```bash
cm init --url http://localhost:8080 --stealth
```

This adds `.checkmate/` to `.git/info/exclude`, keeping it local to your machine.

### Interactive vs Non-Interactive

- **Terminal (TTY)**: `cm init` prompts for base URL interactively
- **Non-TTY (agents)**: Requires `--url` flag with clear error message

## Test Specifications

Test specs are YAML files in `.checkmate/tests/`:

```yaml
name: "User API Tests"
description: "Test user CRUD operations"

env:
  base_url: "http://localhost:8080"  # Optional, uses config if not set
  timeout_ms: 5000

requests:
  create_user:
    body:
      name: "Test User"
      email: "test@example.com"
    headers:
      Authorization: "Bearer ${TOKEN}"  # ${VAR} works in headers, bodies, endpoints, query params

tests:
  user_creation:
    description: "Create a new user"
    endpoint: /api/users
    method: POST
    requests: [create_user]
    expect_status: 201
    assertions:
      - query: "$[id]"
        expect_type: string
        message: "User ID should be returned"
      - query: "$[name]"
        expect: "Test User"
```

### Assertion Types

```yaml
assertions:
  # Exact value match
  - query: "$[doc][status]"
    expect: "active"

  # Type checking
  - query: "$[doc][count]"
    expect_type: number  # string, number, boolean, array, object, null

  # Existence check
  - query: "$[doc][field]?"
    expect: true

  # Numeric comparisons
  - query: "$[doc][count]"
    expect_gte: 1
    expect_lte: 100

  # Array operations
  - query: "$[doc][items].length()"
    expect_gte: 1

  # Compare to previous response
  - query: "$[doc][counter]"
    expect_gt: "$[@prev][counter]"
```

### Multi-Request Tests

For testing stateful behavior (counters, rate limits, sessions):

```yaml
tests:
  counter_increments:
    description: "Counter should increment on each request"
    endpoint: /api/counter
    requests: [increment, increment, increment]
    skip_first: true  # Don't assert on baseline request
    assertions:
      - query: "$[doc][count]"
        expect_gte: 1
      - query: "$[@prev][count]"
        expect_lt: "$[doc][count]"
        message: "Count should increase"
```

### Variable Extraction

Extract values from responses and inject them into subsequent requests using `extract:` and `{{var}}`:

```yaml
requests:
  login:
    body:
      username: "${USERNAME}"
      password: "${PASSWORD}"
    extract:
      auth_token: "$[token]"
      user_id: "$[user][id]"

  get_profile:
    headers:
      Authorization: "Bearer {{auth_token}}"

tests:
  auth_flow:
    endpoint: /api/login
    method: POST
    requests: [login, get_profile]
    assertions:
      - query: "$[name]"
        expect_type: string
```

`${VAR}` expands environment variables. `{{var}}` injects values extracted from previous responses.

## API Version Diff

Compare responses across API versions to detect structural changes. Diffs use the same YAML spec files and discovery as tests, and runs are recorded to history.

```yaml
env:
  base_url: "http://localhost:8080"

requests:
  login:
    body: { user: "${USER}", pass: "${PASS}" }
    extract:
      token: "$[access_token]"
  auth_get:
    headers:
      Authorization: "Bearer {{token}}"
  v1_query:
    body: { format: "legacy" }
  v2_query:
    body: { format: "modern" }

diffs:
  user_endpoint_versions:
    description: "Compare v1 vs v2 user listing"
    setup:
      - request: login
        endpoint: /auth/login
    method: GET
    endpoints:
      - name: v1
        path: /api/v1/users
      - name: v2
        path: /api/v2/users
    requests:
      - auth_get                      # Sent to all endpoints
      - name: v1_query
        scope: [v1]                   # Only sent to v1
      - name: v2_query
        scope: [v2]                   # Only sent to v2
    assertions:
      - query: "$[additions]"
        expect: 0
        message: "No new fields expected"
      - query: "$[changes].any(@[change_type] == 'removed')"
        expect: false
        message: "No fields should be removed"
```

### Setup (Authentication)

Run auth/login requests before endpoint iteration. Extracted variables persist into diff requests:

```yaml
setup:
  - request: login          # Request name from the requests map
    endpoint: /auth/login   # Own endpoint (not from the endpoints list)
    method: POST            # Defaults to POST
```

### Named Endpoints

Give endpoints a name for use with scoped requests and assertions. Simple strings and named objects can be mixed:

```yaml
endpoints:
  - /api/v1/users                # name defaults to the path
  - name: v2
    path: /api/v2/users
```

### Scoped Requests

Different API versions often need different payloads. Scope requests to specific endpoints by name:

```yaml
requests:
  - common_request                # Unscoped — sent to all endpoints
  - name: v1_only
    scope: [v1]                   # Only sent to the v1 endpoint
  - name: v2_only
    scope: [v2]                   # Only sent to the v2 endpoint
```

### Scoped Assertions

When comparing 3+ endpoints, scope assertions to specific pairs:

```yaml
assertions:
  - query: "$[removals]"
    expect: 0                     # Applies to all pairs
  - query: "$[additions]"
    expect: 0
    scope: [v1, v2]              # Only for the v1↔v2 comparison
```

### Diff Result Document

Assertions query against a diff result with this structure:

```json
{
  "changes": [
    { "change_type": "added", "path": "total_pages", "to": 1 },
    { "change_type": "removed", "path": "legacy_flag", "from": true },
    { "change_type": "type_changed", "path": "count", "from": "string", "to": "number" },
    { "change_type": "value_changed", "path": "role", "from": "user", "to": "admin" }
  ],
  "additions": 1,
  "removals": 1,
  "type_changes": 1,
  "value_changes": 1,
  "base_response": { ... },
  "target_response": { ... }
}
```

### Running Diffs

```bash
# Run all diffs
cm diff run

# Run specific spec
cm diff run api_versions

# Run specific diff by name
cm diff run --diff user_endpoint_versions

# List available diffs
cm diff list
```

**Output modes**: In a TTY (interactive terminal), diffs display with colored structural output. In non-TTY (piped to scripts/agents), diffs output structured JSON. Diff runs are recorded to history — view with `cm history`.

## Running Tests

### Run All Tests

```bash
cm test run
```

Auto-discovers all `.yaml` files in `.checkmate/tests/`.

### Run Specific Specs

```bash
# By name (resolves to .checkmate/tests/users.yaml)
cm test run users

# By path
cm test run ./custom/path/test.yaml

# Multiple specs
cm test run users orders payments
```

### Run Specific Test

```bash
cm test run users --test user_creation
```

### Verbose Output

```bash
cm test run -v
```

### Other Commands

```bash
# List all tests
cm test list

# Validate specs without running
cm test validate
```

## Configuration

### Hierarchy (lowest to highest priority)

1. **Built-in defaults**
2. **User config**: `~/.config/checkmate/config.toml`
3. **Project config**: `.checkmate/config.toml`
4. **Environment variables**: `CM_*`

### Project Configuration

`.checkmate/config.toml`:
```toml
[env]
base_url = "http://localhost:8080"
timeout_ms = 5000

[defaults]
fail_fast = false
expect_status = 200
```

### Environment Variables

Override any setting with `CM_` prefix and double underscore for nesting:

```bash
CM_ENV__BASE_URL="http://staging:8080" cm test run
CM_ENV__TIMEOUT_MS=10000 cm test run
CM_DEFAULTS__FAIL_FAST=true cm test run
```

### View Configuration

```bash
# Show current config
cm config show

# Show as JSON
cm config show --json

# Show config sources
cm config sources
```

## Test History

Every test run is recorded with git context in `.checkmate/runs/runs.jsonl`.

### View History

```bash
# Recent runs
cm history

# Filter by spec
cm history --spec users

# Filter by commit
cm history --commit abc123

# More results
cm history -n 20
```

### View Run Details

```bash
cm show cm-run-a3f
```

Output:
```
Run: cm-run-a3f
Time: 2024-01-15T10:30:00Z
Spec: users.yaml
Results: 5/5 passed, 0 failed, 0 errors (1234ms)

Git Context:
  Commit: abc1234
  Branch: feature/user-api
  Dirty: no
  Message: Add user validation
```

### History Format (JSONL)

Each line in `runs.jsonl` is a complete run record:
```json
{
  "id": "cm-run-a3f",
  "timestamp": "2024-01-15T10:30:00Z",
  "git": {
    "commit": "abc1234",
    "branch": "feature/user-api",
    "dirty": false,
    "message": "Add user validation"
  },
  "summary": {
    "total": 5,
    "passed": 5,
    "failed": 0,
    "errors": 0,
    "duration_ms": 1234
  },
  "spec_file": "users.yaml"
}
```

## Hooks

Run custom scripts at test lifecycle events. Place executable scripts in `.checkmate/hooks/`:

| Hook | When | Use Case |
|------|------|----------|
| `pre_run` | Before tests start | Setup, notifications |
| `post_run` | After tests complete | Cleanup, reporting |
| `on_pass` | All tests passed | Success notifications |
| `on_fail` | Any test failed | Alerts, CI integration |

### Environment Variables

Hooks receive context via environment:

| Variable | Description |
|----------|-------------|
| `CM_RUN_ID` | Run identifier (e.g., `cm-run-a3f`) |
| `CM_SPEC` | Spec file name |
| `CM_TOTAL` | Total test count |
| `CM_PASSED` | Passed count |
| `CM_FAILED` | Failed count |
| `CM_ERRORS` | Error count |
| `CM_DURATION_MS` | Duration in milliseconds |

### Example Hook

`.checkmate/hooks/on_fail`:
```bash
#!/bin/bash
echo "Tests failed! Run: $CM_RUN_ID"
echo "Results: $CM_PASSED/$CM_TOTAL passed"
# Send Slack notification, create issue, etc.
```

Make executable: `chmod +x .checkmate/hooks/on_fail`

## Clove Query Language

Checkmate uses Clove for JSON assertions. Key features:

### Scopes

- `$[doc]` - Current response body
- `$[@prev]` - Previous response (for multi-request tests)

### Navigation

```
$[doc][user][name]           # Nested object
$[doc][items][0]             # Array index
$[doc][items][0][id]         # Nested in array
```

### Methods

```
$[doc][items].length()       # Array length
$[doc][name].upper()         # String uppercase
$[doc][price].round()        # Number rounding
```

### Existence Check

```
$[doc][optional]?            # Returns true/false
```

For full documentation: `cm doc clove`

## CLI Reference

### Core Commands

| Command | Description |
|---------|-------------|
| `cm init` | Initialize Checkmate in current directory |
| `cm test run` | Run test specifications |
| `cm test list` | List available tests |
| `cm test validate` | Validate specs without running |
| `cm diff run` | Run diff specifications |
| `cm diff list` | List available diffs |
| `cm history` | Show test run history |
| `cm show <id>` | Show details of a specific run |
| `cm config show` | Display current configuration |

### Documentation

| Command | Description |
|---------|-------------|
| `cm docs` | Full documentation overview |
| `cm doc <category>` | Category-specific docs |
| `cm onboard` | Quick-start guide for AI agents |
| `cm prime` | Output project context (for hooks) |

### Categories for `cm doc`

- `assertions` - Assertion types and examples
- `requests` - Request definition and variable extraction
- `env` - Environment variables and configuration
- `diff` - API version diff comparison
- `scopes` - Clove query scopes
- `examples` - Complete examples
- `cli` - CLI reference

## AI Agent Integration

### Claude Plugin

Install the Checkmate Claude plugin to get automatic context injection:

```bash
# Plugin auto-runs 'cm prime' on session start
# Provides: test specs, recent runs, quick reference
```

### Agent Onboarding

```bash
cm onboard
```

Outputs a condensed guide for AI agents covering:
- Test spec format
- Running tests
- Interpreting results
- Common patterns

### AGENTS.md

Add to your project's `AGENTS.md`:
```markdown
## API Testing

This project uses Checkmate for API testing.

- Run tests: `cm test run`
- Run diffs: `cm diff run`
- View history: `cm history`
- Documentation: `cm docs`
```

## Project Structure

```
your-project/
├── .checkmate/
│   ├── config.toml      # Project settings
│   ├── tests/           # Test specifications
│   │   ├── users.yaml
│   │   ├── orders.yaml
│   │   └── ...
│   ├── runs/            # Test history
│   │   ├── runs.jsonl   # Run records
│   │   └── .gitignore   # Ignores JSONL by default
│   └── hooks/           # Lifecycle scripts
│       ├── pre_run
│       ├── on_pass
│       └── on_fail
└── ...
```

## Best Practices

1. **One concern per spec file** - Group related tests together
2. **Use descriptive test names** - `user_creation_with_valid_data` not `test1`
3. **Include failure messages** - Help debugging when assertions fail
4. **Commit test specs** - Track them alongside your code
5. **Use stealth mode** - For personal testing on shared projects
6. **Review history** - Track test stability across commits

## Troubleshooting

### "No .checkmate/ found"

Run `cm init --url <your-api-url>` in your project root.

### Tests not discovered

Ensure specs are in `.checkmate/tests/` with `.yaml` or `.yml` extension.

### Config not applied

Check priority: env vars > project config > user config > defaults.
Use `cm config sources` to see what's being loaded.

### Hooks not running

1. Check file exists in `.checkmate/hooks/`
2. Ensure it's executable: `chmod +x .checkmate/hooks/on_pass`
3. Check script has correct shebang: `#!/bin/bash`

## License

MIT