checkmate-cli 0.4.1

# Checkmate Test Specification Format

This document defines the YAML-based test specification format for the Checkmate API testing framework.

## Overview

Test specifications are YAML files that define:
- Reusable request templates
- Test cases with assertions
- Environment configuration

The test runner is **stateless** but provides access to the previous response within a test sequence via the `@prev` scope reference.

## File Structure

```yaml
# Optional metadata
name: "Test Suite Name"
description: "Suite description"

# Environment configuration
env:
  base_url: "${BASE_URL:-http://localhost:80}"
  timeout_ms: 5000

# Reusable request definitions
requests:
  request_name:
    body:
      field: value
    headers:
      Content-Type: application/json

# Test definitions
tests:
  test_name:
    endpoint: /api/v1/endpoint
    method: POST  # default
    requests: [request_name]
    assertions:
      - query: "$[field]"
        expect: value
```

## Request Definitions

Requests are reusable templates that can be referenced by multiple tests.

```yaml
requests:
  # Simple request with body
  order_john:
    body:
      name: John Doe
      email: john@example.com
      ip: 192.168.1.1
      user_agent: "Mozilla/5.0"

  # Request with headers
  authenticated_request:
    headers:
      Authorization: "Bearer ${TOKEN}"
    body:
      data: value

  # Request inheriting from another
  order_jane:
    extends: order_john
    body:
      name: Jane Doe
      email: jane@example.com
```

### Request Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `body` | object | No | Request body (JSON) |
| `headers` | object | No | HTTP headers |
| `extends` | string | No | Inherit from another request |
| `query_params` | object | No | URL query parameters |

## Test Definitions

Tests define a sequence of requests and assertions.

```yaml
tests:
  counter_increment_test:
    description: "Verify counters increment on repeated requests"
    endpoint: /api/v1/check-order
    method: POST
    requests: [order_john, order_john, order_john]
    skip_first: true
    fail_fast: true
    assertions:
      - query: "$[securely][metrics][purchase_volume][by_ip][by_minutes][1][current]"
        expect_gt: "@prev[securely][metrics][purchase_volume][by_ip][by_minutes][1][current]"
```

### Test Fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `description` | string | - | Human-readable test description |
| `endpoint` | string | **required** | API endpoint path |
| `method` | string | `POST` | HTTP method |
| `requests` | list | **required** | Request names to execute in sequence |
| `skip_first` | boolean | `false` | Skip assertions on first request (for baseline) |
| `fail_fast` | boolean | `false` | Stop test on first assertion failure |
| `assertions` | list | **required** | Assertion definitions |
| `expect_status` | integer | `200` | Expected HTTP status code |
| `timeout_ms` | integer | env value | Per-test timeout override |

## Assertions

Assertions use Clove queries to validate response data.

### Basic Value Assertion

```yaml
assertions:
  - query: "$[cm][decision][blocked]"
    expect: false
```

### Comparison Assertions

```yaml
assertions:
  # Greater than
  - query: "$[counter]"
    expect_gt: 0

  # Less than
  - query: "$[score]"
    expect_lt: 100

  # Greater than or equal
  - query: "$[count]"
    expect_gte: 1

  # Less than or equal
  - query: "$[risk]"
    expect_lte: 0.5
```

### Previous Response Comparison

The `@prev` scope provides access to the previous response in a multi-request test.

```yaml
assertions:
  # Current value > previous value
  - query: "$[metrics][count]"
    expect_gt: "@prev[metrics][count]"

  # Compare nested values
  - query: "$[securely][metrics][unique_counts][by_email][by_hours][1][current]"
    expect_gt: "@prev[securely][metrics][unique_counts][by_email][by_hours][1][current]"
```

### Existence Assertions

```yaml
assertions:
  # Field must exist
  - query: "$[response][data]?"
    expect: true

  # Field must not exist
  - query: "$[error]?"
    expect: false
```

### Array Assertions

```yaml
assertions:
  # Array length
  - query: "$[items].length()"
    expect_gte: 1

  # Array contains value
  - query: "$[tags].contains('fraud')"
    expect: true

  # Check last element (negative index)
  - query: "$[events][-1][type]"
    expect: "completed"
```

### Type Assertions

```yaml
assertions:
  - query: "$[data]"
    expect_type: object

  - query: "$[count]"
    expect_type: integer

  - query: "$[items]"
    expect_type: array
```

### HTTP Status Assertions

```yaml
tests:
  validation_error_test:
    endpoint: /api/v1/check-order
    requests: [invalid_request]
    expect_status: 422
    assertions:
      - query: "$[error]"
        expect_type: string
```

### Assertion Fields

| Field | Type | Description |
|-------|------|-------------|
| `query` | string | Clove query to evaluate |
| `expect` | any | Exact value match |
| `expect_gt` | number/query | Greater than |
| `expect_lt` | number/query | Less than |
| `expect_gte` | number/query | Greater than or equal |
| `expect_lte` | number/query | Less than or equal |
| `expect_type` | string | Type check (string, integer, number, boolean, array, object, null) |
| `expect_match` | string | Regex pattern match |
| `message` | string | Custom failure message |

## Environment Variables

Environment variables can be used in any string value using `${VAR}` syntax.

```yaml
env:
  base_url: "${BASE_URL:-http://localhost:80}"
  api_key: "${API_KEY}"

requests:
  authenticated:
    headers:
      Authorization: "Bearer ${API_KEY}"
```

### Default Values

Use `${VAR:-default}` syntax for defaults:

```yaml
env:
  timeout_ms: "${TIMEOUT:-5000}"
  base_url: "${BASE_URL:-http://localhost:8080}"
```

## Scope References

Checkmate uses Clove's scope reference syntax for accessing data:

| Scope | Description |
|-------|-------------|
| `$` | Current response root |
| `@prev` | Previous response in sequence |
| `@env` | Environment variables |
| `@req` | Current request data |

### Examples

```yaml
assertions:
  # Current response
  - query: "$[data][id]"
    expect_type: string

  # Previous response comparison
  - query: "$[counter]"
    expect_gt: "@prev[counter]"

  # Environment value
  - query: "$[environment]"
    expect: "@env[EXPECTED_ENV]"
```

## Complete Example

```yaml
name: "Fraud Detection API Tests"
description: "Tests for the check-order endpoint"

env:
  base_url: "${BASE_URL:-http://localhost:80}"
  timeout_ms: 5000

requests:
  order_john:
    body:
      name: John Doe
      email: john@example.com
      ip: 192.168.1.100
      user_agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0"

  order_jane:
    body:
      name: Jane Smith
      email: jane@example.com
      ip: 192.168.1.101
      user_agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Safari/17.0"

  invalid_order:
    body:
      name: ""
      email: "not-an-email"

tests:
  basic_response_structure:
    description: "Verify response contains expected structure"
    endpoint: /api/v1/check-order
    requests: [order_john]
    assertions:
      - query: "$[cm][decision]?"
        expect: true
      - query: "$[securely][metrics]?"
        expect: true
      - query: "$[cm][decision][blocked]"
        expect_type: boolean

  counter_increments:
    description: "Verify purchase counters increment on repeated requests"
    endpoint: /api/v1/check-order
    requests: [order_john, order_john, order_john, order_john]
    skip_first: true
    fail_fast: true
    assertions:
      - query: "$[securely][metrics][purchase_volume][by_ip][by_minutes][1][current]"
        expect_gt: "@prev[securely][metrics][purchase_volume][by_ip][by_minutes][1][current]"
      - query: "$[securely][metrics][purchase_volume][by_email][by_minutes][1][current]"
        expect_gt: "@prev[securely][metrics][purchase_volume][by_email][by_minutes][1][current]"

  unique_counts_tracking:
    description: "Verify unique email/IP tracking"
    endpoint: /api/v1/check-order
    requests: [order_john, order_jane]
    skip_first: true
    assertions:
      - query: "$[securely][metrics][unique_counts][by_ip][by_hours][1][current]"
        expect_gte: "@prev[securely][metrics][unique_counts][by_ip][by_hours][1][current]"

  anomaly_detection_arrays:
    description: "Verify anomaly detection returns proper arrays"
    endpoint: /api/v1/check-order
    requests: [order_john]
    assertions:
      - query: "$[securely][anomaly_detection][normal]"
        expect_type: array
      - query: "$[securely][anomaly_detection][abnormal]"
        expect_type: array

  validation_error_handling:
    description: "Verify invalid requests return proper errors"
    endpoint: /api/v1/check-order
    requests: [invalid_order]
    expect_status: 422
    assertions:
      - query: "$[detail]"
        expect_type: array

  response_time:
    description: "Verify response time is acceptable"
    endpoint: /api/v1/check-order
    requests: [order_john]
    timeout_ms: 1000
    assertions:
      - query: "$[cm][decision]?"
        expect: true
```

## CLI Usage

```bash
# Run all tests in a spec file
cm test run tests/fraud_api.yaml

# Run specific test
cm test run tests/fraud_api.yaml --test counter_increments

# Validate spec without running
cm test validate tests/fraud_api.yaml

# Run with verbose output
cm test run tests/fraud_api.yaml --verbose

# Run with custom environment
BASE_URL=http://staging:8080 cm test run tests/fraud_api.yaml
```

## Report Output

Test results are output in JSON format by default:

```json
{
  "suite": "Fraud Detection API Tests",
  "timestamp": "2024-01-15T10:30:00Z",
  "duration_ms": 1234,
  "summary": {
    "total": 6,
    "passed": 5,
    "failed": 1,
    "skipped": 0
  },
  "tests": [
    {
      "name": "basic_response_structure",
      "status": "passed",
      "duration_ms": 150,
      "requests": 1,
      "assertions": {
        "total": 3,
        "passed": 3,
        "failed": 0
      }
    },
    {
      "name": "counter_increments",
      "status": "failed",
      "duration_ms": 450,
      "requests": 4,
      "assertions": {
        "total": 2,
        "passed": 1,
        "failed": 1
      },
      "failures": [
        {
          "request_index": 2,
          "assertion": "$[securely][metrics][purchase_volume][by_email][by_minutes][1][current] > @prev[...]",
          "expected": "> 5",
          "actual": 5,
          "message": "Counter did not increment"
        }
      ]
    }
  ]
}
```

### Report Formats

```bash
# JSON (default)
cm report generate --format json

# YAML
cm report generate --format yaml

# Markdown
cm report generate --format md

# HTML
cm report generate --format html
```