# Test Specification Format
Complete reference for Checkmate test specification YAML format.
## File Location
Test specs are YAML files in `.checkmate/tests/`:
```
.checkmate/
└── tests/
├── users.yaml
├── orders.yaml
└── health.yaml
```
## Basic Structure
```yaml
name: "Suite Name"
description: "What this suite tests"
env:
base_url: "http://localhost:8080"
timeout_ms: 5000
requests:
request_name:
body: {}
headers: {}
tests:
test_name:
description: "What this test verifies"
endpoint: /api/path
method: GET
requests: [request_name]
expect_status: 200
assertions: []
```
## Top-Level Fields
| `name` | string | No | Suite name (shown in output) |
| `description` | string | No | Suite description |
| `env` | object | No | Environment settings |
| `requests` | object | No | Named request definitions |
| `tests` | object | Yes | Test case definitions |
---
## env Section
Override global configuration for this spec.
```yaml
env:
base_url: "http://localhost:8080"
timeout_ms: 5000
```
| `base_url` | string | From config | API base URL |
| `timeout_ms` | integer | 5000 | Request timeout in milliseconds |
If not specified, values are taken from project configuration.
---
## requests Section
Define reusable request templates.
```yaml
requests:
create_user:
body:
name: "Test User"
email: "test@example.com"
headers:
Content-Type: "application/json"
Authorization: "Bearer token123"
admin_request:
headers:
Authorization: "Bearer admin-token"
body:
role: "admin"
```
### Request Fields
| `body` | object | JSON request body |
| `headers` | object | HTTP headers |
### Using Requests in Tests
Reference by name in the `requests` array:
```yaml
tests:
user_creation:
endpoint: /api/users
method: POST
requests: [create_user]
```
---
## tests Section
Define test cases.
```yaml
tests:
test_name:
description: "What this test verifies"
endpoint: /api/users
method: POST
requests: [request_name]
expect_status: 201
fail_fast: false
skip_first: false
timeout_ms: 10000
assertions:
- query: "$[id]"
expect_type: string
```
### Test Fields
| `description` | string | - | Test description |
| `endpoint` | string | Required | API endpoint path |
| `method` | string | GET | HTTP method |
| `requests` | array | Required | Request names to execute |
| `expect_status` | integer | 200 | Expected HTTP status code |
| `fail_fast` | boolean | false | Stop on first assertion failure |
| `skip_first` | boolean | false | Skip assertions on first request |
| `timeout_ms` | integer | From env | Override timeout for this test |
| `assertions` | array | [] | Assertion definitions |
### HTTP Methods
Supported methods: `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, `HEAD`, `OPTIONS`
```yaml
tests:
get_users:
endpoint: /api/users
method: GET
requests: [empty]
create_user:
endpoint: /api/users
method: POST
requests: [new_user]
update_user:
endpoint: /api/users/123
method: PUT
requests: [updated_user]
delete_user:
endpoint: /api/users/123
method: DELETE
requests: [empty]
```
---
## Multi-Request Tests
Execute multiple requests in sequence to test stateful behavior.
### Basic Multi-Request
```yaml
tests:
counter_increments:
endpoint: /api/counter
method: POST
requests: [increment, increment, increment]
assertions:
- query: "$[count]"
expect_gte: 1
```
### Using skip_first
Skip assertions on the first request (useful for establishing baseline):
```yaml
tests:
rate_limit_test:
endpoint: /api/limited
requests: [request, request, request, request, request]
skip_first: true
assertions:
- query: "$[remaining]"
expect_lt: "$[@prev][remaining]"
```
### Comparing to Previous Response
Use `$[@prev]` scope to compare current response to previous:
```yaml
assertions:
- query: "$[counter]"
expect_gt: "$[@prev][counter]"
message: "Counter should increment"
```
---
## Assertions Section
See [ASSERTIONS.md](ASSERTIONS.md) for complete assertion reference.
### Quick Reference
```yaml
assertions:
# Exact match
- query: "$[status]"
expect: "active"
# Type check
- query: "$[id]"
expect_type: string
# Existence
- query: "$[field]?"
expect: true
# Comparisons
- query: "$[count]"
expect_gte: 1
expect_lte: 100
# With message
- query: "$[valid]"
expect: true
message: "Response should be valid"
# Array at root
- query: "$.length()"
expect_gte: 0
```
---
## Complete Example
```yaml
name: "User API Tests"
description: "Test user CRUD operations"
env:
base_url: "http://localhost:8080"
timeout_ms: 5000
requests:
valid_user:
body:
name: "John Doe"
email: "john@example.com"
role: "user"
headers:
Content-Type: "application/json"
admin_user:
body:
name: "Admin"
email: "admin@example.com"
role: "admin"
headers:
Content-Type: "application/json"
X-Admin-Key: "secret123"
empty:
body: {}
tests:
create_user_success:
description: "Create a new user with valid data"
endpoint: /api/users
method: POST
requests: [valid_user]
expect_status: 201
assertions:
- query: "$[id]"
expect_type: string
message: "Should return user ID"
- query: "$[name]"
expect: "John Doe"
- query: "$[email]"
expect: "john@example.com"
- query: "$[created_at]?"
expect: true
message: "Should have created_at timestamp"
create_user_invalid:
description: "Reject user with invalid email"
endpoint: /api/users
method: POST
requests: [empty]
expect_status: 422
assertions:
- query: "$[error]?"
expect: true
- query: "$[error][code]"
expect: "VALIDATION_ERROR"
list_users:
description: "List all users"
endpoint: /api/users
method: GET
requests: [empty]
expect_status: 200
assertions:
- query: "$.length()"
expect_gte: 0
message: "Should return array"
admin_access:
description: "Admin can access protected endpoint"
endpoint: /api/admin/stats
method: GET
requests: [admin_user]
expect_status: 200
assertions:
- query: "$[user_count]"
expect_type: number
```
---
## Best Practices
### 1. One Concern Per Spec
Group related tests in a single file:
- `users.yaml` - User CRUD
- `auth.yaml` - Authentication flows
- `orders.yaml` - Order operations
### 2. Descriptive Names
```yaml
# Good
tests:
user_creation_with_valid_email:
user_creation_rejects_duplicate_email:
# Avoid
tests:
test1:
test2:
```
### 3. Include Failure Messages
```yaml
assertions:
- query: "$[status]"
expect: "active"
message: "User should be active after creation"
```
### 4. Use Request Templates
DRY - define once, use many times:
```yaml
requests:
auth_header:
headers:
Authorization: "Bearer ${TOKEN}"
tests:
protected_endpoint_1:
requests: [auth_header]
protected_endpoint_2:
requests: [auth_header]
```
### 5. Test Error Cases
Don't just test happy paths:
```yaml
tests:
invalid_input:
expect_status: 422
unauthorized:
expect_status: 401
not_found:
expect_status: 404
```
### 6. Handle Array Responses
When list endpoints return arrays directly:
```yaml
# GET /items returns: [{"id": 1}, {"id": 2}]
tests:
list_items:
endpoint: /items
method: GET
requests: [empty]
assertions:
- query: "$.length()"
expect_gte: 0
```
---
## See Also
- [Assertions](ASSERTIONS.md) - All assertion types
- [Clove Queries](CLOVE.md) - Query language reference
- [Quick Start](QUICKSTART.md) - Get started fast