checkmate-cli 0.4.1

Checkmate - API Testing Framework CLI
checkmate-cli-0.4.1 is not a library.

Checkmate

AI-native API testing framework with YAML specs and git-connected history.

Checkmate makes API testing simple for both humans and AI agents. Write declarative test specs in YAML, run assertions with Clove queries, and track results with full git context.

Quick Start

# Build from source
cargo build --release

# Initialize in your project
cm init --url http://localhost:8080

# Create a test spec
cat > .checkmate/tests/health.yaml << 'EOF'
name: "Health Check"
tests:
  api_responds:
    endpoint: /health
    assertions:
      - query: "$[doc][status]"
        expect: "ok"
EOF

# Run tests
cm test run

Features

  • YAML Test Specs: Declarative, readable test definitions
  • Clove Query Language: Powerful JSON assertions with $[field] and $[@prev] scopes
  • API Version Diff: Compare responses across endpoints to detect structural changes (cm diff run)
  • Variable Extraction: Extract values from responses for multi-step auth flows
  • Universal Env Vars: ${VAR} expansion in endpoints, headers, bodies, and query params
  • Git-Connected History: Every test run captures commit, branch, and dirty status
  • Auto-Discovery: Run cm test run from any subdirectory
  • Configuration Hierarchy: Defaults → User config → Project config → Environment
  • Hook System: Run custom scripts on test lifecycle events
  • AI-Native: Built-in documentation, Claude plugin, and agent-friendly CLI

Installation

From Source

git clone https://github.com/your-org/checkmate
cd checkmate
cargo build --release
# Binary at target/release/cm

Add to PATH

export PATH="$PATH:/path/to/checkmate/target/release"

Initialization

Standard Mode

cm init --url http://localhost:8080

Creates .checkmate/ directory structure:

.checkmate/
├── config.toml    # Project configuration
├── tests/         # Test specifications
├── runs/          # Test run history (JSONL)
└── hooks/         # Custom hook scripts

Stealth Mode

For personal use on shared projects without committing Checkmate files:

cm init --url http://localhost:8080 --stealth

This adds .checkmate/ to .git/info/exclude, keeping it local to your machine.

Interactive vs Non-Interactive

  • Terminal (TTY): cm init prompts for base URL interactively
  • Non-TTY (agents): Requires --url flag with clear error message

Test Specifications

Test specs are YAML files in .checkmate/tests/:

name: "User API Tests"
description: "Test user CRUD operations"

env:
  base_url: "http://localhost:8080"  # Optional, uses config if not set
  timeout_ms: 5000

requests:
  create_user:
    body:
      name: "Test User"
      email: "test@example.com"
    headers:
      Authorization: "Bearer ${TOKEN}"  # ${VAR} works in headers, bodies, endpoints, query params

tests:
  user_creation:
    description: "Create a new user"
    endpoint: /api/users
    method: POST
    requests: [create_user]
    expect_status: 201
    assertions:
      - query: "$[id]"
        expect_type: string
        message: "User ID should be returned"
      - query: "$[name]"
        expect: "Test User"

Assertion Types

assertions:
  # Exact value match
  - query: "$[doc][status]"
    expect: "active"

  # Type checking
  - query: "$[doc][count]"
    expect_type: number  # string, number, boolean, array, object, null

  # Existence check
  - query: "$[doc][field]?"
    expect: true

  # Numeric comparisons
  - query: "$[doc][count]"
    expect_gte: 1
    expect_lte: 100

  # Array operations
  - query: "$[doc][items].length()"
    expect_gte: 1

  # Compare to previous response
  - query: "$[doc][counter]"
    expect_gt: "$[@prev][counter]"

Multi-Request Tests

For testing stateful behavior (counters, rate limits, sessions):

tests:
  counter_increments:
    description: "Counter should increment on each request"
    endpoint: /api/counter
    requests: [increment, increment, increment]
    skip_first: true  # Don't assert on baseline request
    assertions:
      - query: "$[doc][count]"
        expect_gte: 1
      - query: "$[@prev][count]"
        expect_lt: "$[doc][count]"
        message: "Count should increase"

Variable Extraction

Extract values from responses and inject them into subsequent requests using extract: and {{var}}:

requests:
  login:
    body:
      username: "${USERNAME}"
      password: "${PASSWORD}"
    extract:
      auth_token: "$[token]"
      user_id: "$[user][id]"

  get_profile:
    headers:
      Authorization: "Bearer {{auth_token}}"

tests:
  auth_flow:
    endpoint: /api/login
    method: POST
    requests: [login, get_profile]
    assertions:
      - query: "$[name]"
        expect_type: string

${VAR} expands environment variables. {{var}} injects values extracted from previous responses.

API Version Diff

Compare responses across API versions to detect structural changes. Diffs use the same YAML spec files and discovery as tests, and runs are recorded to history.

env:
  base_url: "http://localhost:8080"

requests:
  login:
    body: { user: "${USER}", pass: "${PASS}" }
    extract:
      token: "$[access_token]"
  auth_get:
    headers:
      Authorization: "Bearer {{token}}"
  v1_query:
    body: { format: "legacy" }
  v2_query:
    body: { format: "modern" }

diffs:
  user_endpoint_versions:
    description: "Compare v1 vs v2 user listing"
    setup:
      - request: login
        endpoint: /auth/login
    method: GET
    endpoints:
      - name: v1
        path: /api/v1/users
      - name: v2
        path: /api/v2/users
    requests:
      - auth_get                      # Sent to all endpoints
      - name: v1_query
        scope: [v1]                   # Only sent to v1
      - name: v2_query
        scope: [v2]                   # Only sent to v2
    assertions:
      - query: "$[additions]"
        expect: 0
        message: "No new fields expected"
      - query: "$[changes].any(@[change_type] == 'removed')"
        expect: false
        message: "No fields should be removed"

Setup (Authentication)

Run auth/login requests before endpoint iteration. Extracted variables persist into diff requests:

setup:
  - request: login          # Request name from the requests map
    endpoint: /auth/login   # Own endpoint (not from the endpoints list)
    method: POST            # Defaults to POST

Named Endpoints

Give endpoints a name for use with scoped requests and assertions. Simple strings and named objects can be mixed:

endpoints:
  - /api/v1/users                # name defaults to the path
  - name: v2
    path: /api/v2/users

Scoped Requests

Different API versions often need different payloads. Scope requests to specific endpoints by name:

requests:
  - common_request                # Unscoped — sent to all endpoints
  - name: v1_only
    scope: [v1]                   # Only sent to the v1 endpoint
  - name: v2_only
    scope: [v2]                   # Only sent to the v2 endpoint

Scoped Assertions

When comparing 3+ endpoints, scope assertions to specific pairs:

assertions:
  - query: "$[removals]"
    expect: 0                     # Applies to all pairs
  - query: "$[additions]"
    expect: 0
    scope: [v1, v2]              # Only for the v1↔v2 comparison

Diff Result Document

Assertions query against a diff result with this structure:

{
  "changes": [
    { "change_type": "added", "path": "total_pages", "to": 1 },
    { "change_type": "removed", "path": "legacy_flag", "from": true },
    { "change_type": "type_changed", "path": "count", "from": "string", "to": "number" },
    { "change_type": "value_changed", "path": "role", "from": "user", "to": "admin" }
  ],
  "additions": 1,
  "removals": 1,
  "type_changes": 1,
  "value_changes": 1,
  "base_response": { ... },
  "target_response": { ... }
}

Running Diffs

# Run all diffs
cm diff run

# Run specific spec
cm diff run api_versions

# Run specific diff by name
cm diff run --diff user_endpoint_versions

# List available diffs
cm diff list

Output modes: In a TTY (interactive terminal), diffs display with colored structural output. In non-TTY (piped to scripts/agents), diffs output structured JSON. Diff runs are recorded to history — view with cm history.

Running Tests

Run All Tests

cm test run

Auto-discovers all .yaml files in .checkmate/tests/.

Run Specific Specs

# By name (resolves to .checkmate/tests/users.yaml)
cm test run users

# By path
cm test run ./custom/path/test.yaml

# Multiple specs
cm test run users orders payments

Run Specific Test

cm test run users --test user_creation

Verbose Output

cm test run -v

Other Commands

# List all tests
cm test list

# Validate specs without running
cm test validate

Configuration

Hierarchy (lowest to highest priority)

  1. Built-in defaults
  2. User config: ~/.config/checkmate/config.toml
  3. Project config: .checkmate/config.toml
  4. Environment variables: CM_*

Project Configuration

.checkmate/config.toml:

[env]
base_url = "http://localhost:8080"
timeout_ms = 5000

[defaults]
fail_fast = false
expect_status = 200

Environment Variables

Override any setting with CM_ prefix and double underscore for nesting:

CM_ENV__BASE_URL="http://staging:8080" cm test run
CM_ENV__TIMEOUT_MS=10000 cm test run
CM_DEFAULTS__FAIL_FAST=true cm test run

View Configuration

# Show current config
cm config show

# Show as JSON
cm config show --json

# Show config sources
cm config sources

Test History

Every test run is recorded with git context in .checkmate/runs/runs.jsonl.

View History

# Recent runs
cm history

# Filter by spec
cm history --spec users

# Filter by commit
cm history --commit abc123

# More results
cm history -n 20

View Run Details

cm show cm-run-a3f

Output:

Run: cm-run-a3f
Time: 2024-01-15T10:30:00Z
Spec: users.yaml
Results: 5/5 passed, 0 failed, 0 errors (1234ms)

Git Context:
  Commit: abc1234
  Branch: feature/user-api
  Dirty: no
  Message: Add user validation

History Format (JSONL)

Each line in runs.jsonl is a complete run record:

{
  "id": "cm-run-a3f",
  "timestamp": "2024-01-15T10:30:00Z",
  "git": {
    "commit": "abc1234",
    "branch": "feature/user-api",
    "dirty": false,
    "message": "Add user validation"
  },
  "summary": {
    "total": 5,
    "passed": 5,
    "failed": 0,
    "errors": 0,
    "duration_ms": 1234
  },
  "spec_file": "users.yaml"
}

Hooks

Run custom scripts at test lifecycle events. Place executable scripts in .checkmate/hooks/:

Hook When Use Case
pre_run Before tests start Setup, notifications
post_run After tests complete Cleanup, reporting
on_pass All tests passed Success notifications
on_fail Any test failed Alerts, CI integration

Environment Variables

Hooks receive context via environment:

Variable Description
CM_RUN_ID Run identifier (e.g., cm-run-a3f)
CM_SPEC Spec file name
CM_TOTAL Total test count
CM_PASSED Passed count
CM_FAILED Failed count
CM_ERRORS Error count
CM_DURATION_MS Duration in milliseconds

Example Hook

.checkmate/hooks/on_fail:

#!/bin/bash
echo "Tests failed! Run: $CM_RUN_ID"
echo "Results: $CM_PASSED/$CM_TOTAL passed"
# Send Slack notification, create issue, etc.

Make executable: chmod +x .checkmate/hooks/on_fail

Clove Query Language

Checkmate uses Clove for JSON assertions. Key features:

Scopes

  • $[doc] - Current response body
  • $[@prev] - Previous response (for multi-request tests)

Navigation

$[doc][user][name]           # Nested object
$[doc][items][0]             # Array index
$[doc][items][0][id]         # Nested in array

Methods

$[doc][items].length()       # Array length
$[doc][name].upper()         # String uppercase
$[doc][price].round()        # Number rounding

Existence Check

$[doc][optional]?            # Returns true/false

For full documentation: cm doc clove

CLI Reference

Core Commands

Command Description
cm init Initialize Checkmate in current directory
cm test run Run test specifications
cm test list List available tests
cm test validate Validate specs without running
cm diff run Run diff specifications
cm diff list List available diffs
cm history Show test run history
cm show <id> Show details of a specific run
cm config show Display current configuration

Documentation

Command Description
cm docs Full documentation overview
cm doc <category> Category-specific docs
cm onboard Quick-start guide for AI agents
cm prime Output project context (for hooks)

Categories for cm doc

  • assertions - Assertion types and examples
  • requests - Request definition and variable extraction
  • env - Environment variables and configuration
  • diff - API version diff comparison
  • scopes - Clove query scopes
  • examples - Complete examples
  • cli - CLI reference

AI Agent Integration

Claude Plugin

Install the Checkmate Claude plugin to get automatic context injection:

# Plugin auto-runs 'cm prime' on session start
# Provides: test specs, recent runs, quick reference

Agent Onboarding

cm onboard

Outputs a condensed guide for AI agents covering:

  • Test spec format
  • Running tests
  • Interpreting results
  • Common patterns

AGENTS.md

Add to your project's AGENTS.md:

## API Testing

This project uses Checkmate for API testing.

- Run tests: `cm test run`
- Run diffs: `cm diff run`
- View history: `cm history`
- Documentation: `cm docs`

Project Structure

your-project/
├── .checkmate/
│   ├── config.toml      # Project settings
│   ├── tests/           # Test specifications
│   │   ├── users.yaml
│   │   ├── orders.yaml
│   │   └── ...
│   ├── runs/            # Test history
│   │   ├── runs.jsonl   # Run records
│   │   └── .gitignore   # Ignores JSONL by default
│   └── hooks/           # Lifecycle scripts
│       ├── pre_run
│       ├── on_pass
│       └── on_fail
└── ...

Best Practices

  1. One concern per spec file - Group related tests together
  2. Use descriptive test names - user_creation_with_valid_data not test1
  3. Include failure messages - Help debugging when assertions fail
  4. Commit test specs - Track them alongside your code
  5. Use stealth mode - For personal testing on shared projects
  6. Review history - Track test stability across commits

Troubleshooting

"No .checkmate/ found"

Run cm init --url <your-api-url> in your project root.

Tests not discovered

Ensure specs are in .checkmate/tests/ with .yaml or .yml extension.

Config not applied

Check priority: env vars > project config > user config > defaults. Use cm config sources to see what's being loaded.

Hooks not running

  1. Check file exists in .checkmate/hooks/
  2. Ensure it's executable: chmod +x .checkmate/hooks/on_pass
  3. Check script has correct shebang: #!/bin/bash

License

MIT