trip-test — Contract Testing for MCP Servers
Contract testing and regression safety for Model Context Protocol (MCP) servers. Catch breaking changes before they ship.
The Problem
MCP servers expose tools that AI agents depend on. A schema change breaks every agent that depends on it — silently, with no build-time error or test failure. It surfaces later as a 500 error or refused tool call in production.
Example: Rename a required parameter from limit to max_results.
- Old agents call:
search(q="test", limit=10)→ ❌ fails - No CI warning
- No regression test catches it
- Agents break in production
The Solution
trip-test treats MCP server contracts like REST APIs. Record a baseline, detect breaking changes in CI, fail PRs automatically.
Think of it as Postman + Jest snapshots, but for MCP servers.
Quick Start
Installation
Via Cargo:
Via Homebrew (coming soon):
From source:
Basic Usage
1. Connect to your server and list tools:
Output:
Connecting to: python -m my_mcp_server
Tools:
- search (Search records)
inputs: { q: string, limit: integer, }
- fetch (Fetch a record)
inputs: { id: string, }
2. Record a baseline snapshot:
Interactive prompts guide you to invoke a few tools. Captures tool schemas + responses.
3. Check for regressions:
Replays recorded exchanges. Passes if behavior matches.
Results: 2 passed, 0 failed
4. Detect breaking changes:
Compares contracts. Classifies changes.
🚨 BREAKING CHANGES:
• search: Parameter 'limit' removed (was required: true)
• search: Parameter 'max_results' added (required: true)
⚠️ NON-BREAKING CHANGES:
• search: Description updated
Overall Verdict: BREAKING
CLI Commands
connect
List all tools and their schemas from a server.
call
Invoke a single tool and see the exchange.
Output:
record
Interactively record tool calls into a snapshot file.
Prompts you to select tools and invoke them. Captures input + output. Saves to JSON.
check
Replay a snapshot and verify the server still matches.
Exit code: 0 if all pass, 1 if any fail.
diff
Compare baseline contract vs live server. Classify changes.
Output: BREAKING, NON-BREAKING, or ADDITIVE changes.
Exit code: 0 if safe, 1 if breaking.
Global Flags
--verboseor-v— Enable debug output--helpor-h— Show usage--version— Print version--config <file>— Load settings from TOML file
Configuration
TOML Configuration File
Create trip-test.toml in your project root to avoid repeating flags:
[]
# Path to your baseline snapshot file (relative to repo root)
= "contracts/baseline.trip-test.json"
[]
# Command to start your MCP server
# Can be any command that spawns an MCP server:
# - Python: "python -m my_package"
# - Node.js: "node src/index.js"
# - Rust: "./target/release/my-server"
# - Docker: "docker run my-image"
= "python -m my_mcp_server"
# Timeout for server operations (milliseconds)
= 5000
[]
# Whether to treat additive changes (new tools/params) as warnings
= false
# Whether to treat non-breaking changes (descriptions) as warnings
= false
Then just run:
Configuration Examples
Python server with virtual environment:
[]
= "python -m venv/bin/python -m my_server"
Node.js server with npm script:
[]
= "npm run start:mcp"
Rust server (local development):
[]
= "cargo run --release -- --stdio"
Docker container:
[]
= "docker run --rm my-mcp-server:latest"
With environment variables:
[]
# Unix/Linux/Mac
= "env RUST_LOG=debug ./target/release/server"
# Or use shell directly
= "bash -c 'export RUST_LOG=debug && ./target/release/server'"
Multiple servers (use separate config files):
# test-server-v1.toml
# test-server-v2.toml
Snapshot File Format
Snapshots are JSON files that capture a contract baseline.
Version-control this file. It's your regression baseline.
CI/CD Integration
This section shows how to integrate trip-test into your existing CI/CD pipeline.
GitHub Actions
Basic Setup
Create .github/workflows/trip-test.yml in any MCP server repository:
name: trip-test Contract Check
on:
pull_request:
paths:
- 'src/**'
- 'Cargo.toml'
- '.github/workflows/trip-test.yml'
push:
branches:
jobs:
contract-check:
name: Check MCP Contract
runs-on: ubuntu-latest
steps:
# 1. Checkout code
- uses: actions/checkout@v4
with:
fetch-depth: 0
# 2. Setup Rust (if building Rust server)
- uses: dtolnay/rust-toolchain@stable
- uses: swatinem/rust-cache@v2
# 3. Build server
- name: Build MCP server
run: cargo build --release
# 4. Install trip-test
- name: Install trip-test
run: cargo install trip-test
# 5. Run contract check
- name: Check contract
id: check
run: |
trip-test diff \
--baseline contracts/baseline.trip-test.json \
--server "./target/release/my-server" \
--format json > report.json || true
cat report.json
# 6. Comment on PR
- name: Post PR comment
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const fs = require('fs');
const report = JSON.parse(fs.readFileSync('report.json'));
let comment = '## 🔍 trip-test Contract Check\n\n';
if (report.overall_verdict === 'BREAKING') {
comment += '### ❌ **Breaking Changes Detected**\n\n';
comment += 'The following breaking changes were found:\n\n';
comment += '| Tool | Change | Details |\n';
comment += '|------|--------|----------|\n';
report.breaking_changes.forEach(c => {
comment += `| \`${c.tool_name}\` | ${c.change_type} | ${c.details} |\n`;
});
} else {
comment += '### ✅ **Contract is Safe**\n\n';
comment += 'No breaking changes detected.\n\n';
}
if (report.non_breaking_changes.length > 0) {
comment += '\n### ⚠️ Non-Breaking Changes\n\n';
comment += '| Tool | Change | Details |\n';
comment += '|------|--------|----------|\n';
report.non_breaking_changes.forEach(c => {
comment += `| \`${c.tool_name}\` | ${c.change_type} | ${c.details} |\n`;
});
}
if (report.additive_changes.length > 0) {
comment += '\n### ✨ Additive Changes\n\n';
comment += '| Tool | Change | Details |\n';
comment += '|------|--------|----------|\n';
report.additive_changes.forEach(c => {
comment += `| \`${c.tool_name}\` | ${c.change_type} | ${c.details} |\n`;
});
}
comment += '\n---\n';
comment += `**Verdict:** \`${report.overall_verdict}\`\n`;
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: comment
});
# 7. Fail if breaking changes
- name: Enforce contract safety
if: always()
run: |
if grep -q '"overall_verdict": "BREAKING"' report.json; then
echo "❌ Breaking changes detected! PR check failed."
exit 1
fi
For Python Servers
name: trip-test Contract Check (Python)
on:
jobs:
contract-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
cache: 'pip'
- name: Install dependencies
run: |
pip install -e .
pip install mcp
- name: Install trip-test
run: cargo install trip-test
- name: Check contract
run: |
trip-test diff \
--config trip-test.toml \
--format json > report.json || true
- name: Post result
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const report = JSON.parse(fs.readFileSync('report.json'));
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `## trip-test Check\n\nVerdict: **${report.overall_verdict}**`
});
- name: Enforce safety
run: |
if grep -q '"overall_verdict": "BREAKING"' report.json; then
exit 1
fi
For Node.js Servers
name: trip-test Contract Check (Node.js)
on:
jobs:
contract-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Node
uses: actions/setup-node@v3
with:
node-version: '18'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Build
run: npm run build
- name: Install trip-test
run: cargo install trip-test
- name: Check contract
run: |
trip-test diff \
--config trip-test.toml \
--format json > report.json || true
- name: Post result
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const report = JSON.parse(fs.readFileSync('report.json'));
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `## trip-test Check\n\nVerdict: **${report.overall_verdict}**`
});
- name: Enforce safety
run: |
if grep -q '"overall_verdict": "BREAKING"' report.json; then
exit 1
fi
GitLab CI
contract-check:
image: rust:latest
before_script:
- apt-get update && apt-get install -y python3
- cargo install trip-test
script:
# Build your server (adjust for your build system)
- cargo build --release
# Run contract check
- trip-test diff --config trip-test.toml --format json > report.json || true
# Check for breaking changes
- |
if grep -q '"overall_verdict": "BREAKING"' report.json; then
echo "❌ Breaking changes detected!"
cat report.json
exit 1
else
echo "✅ Contract is safe"
fi
artifacts:
paths:
- report.json
expire_in: 30 days
Generic CI (Jenkins, CircleCI, etc.)
The core pattern is:
# 1. Build server
<your-build-command>
# 2. Install trip-test (if not cached)
# 3. Run check
||
# 4. Parse report
if ; then
fi
# 5. Optional: upload report
<your-artifact-upload>
Integration Checklist
When adding trip-test to an existing MCP server:
- Create
trip-test.tomlwith correct server command - Create
contracts/directory - Record baseline:
trip-test record --server "..." --out contracts/baseline.trip-test.json - Commit
contracts/baseline.trip-test.jsonandtrip-test.toml - Add CI workflow (GitHub Actions / GitLab CI / etc.)
- Test locally:
trip-test diff --config trip-test.toml - Open a test PR with intentional breaking change
- Verify CI fails and posts comment
- Revert test change
- Merge and ship ✅
Example: Adding trip-test to Existing Showpad MCP Server
# 1. Clone server
# 2. Install trip-test
# 3. Create config
# 4. Create contracts directory
# 5. Record baseline (interactively)
# 6. Verify
# 7. Commit
# 8. Add GitHub Actions
# 9. Push and open PR
Troubleshooting CI Integration
CI fails to build server:
- Check that build command in
trip-test.tomlworks locally - Ensure all build dependencies are in CI environment
- For Python: verify venv/dependencies
- For Node.js: verify npm install runs
- For Rust: verify Cargo.toml exists
CI fails to install trip-test:
- Add
rustupif not present:curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh - Or pre-build trip-test as a cached artifact
- Or use:
pip install trip-test(if available)
Report not found:
- Check that server command in config works
- Verify baseline snapshot exists:
contracts/baseline.trip-test.json - Run locally first:
trip-test diff --config trip-test.toml
PR comment not posting:
- Ensure GitHub token has
issues:writepermission - Check that issue/PR number is correct
- Verify repository is not private (or token has private repo access)
Testing Your MCP Server
Complete Testing Workflow
This section walks you through a complete testing setup from scratch.
Step 1: Installation
# Clone your MCP server repository
# Install trip-test
# Or, if you have Homebrew:
Step 2: Create Project Structure
# Create a contracts directory for snapshots
# Create initial trip-test config
Step 3: Test Server Connection
Verify your server works:
# List all tools exposed by your server
Expected output:
Connecting to: python -m my_mcp_server
Tools:
- search (Search records)
inputs: { q: string, limit: integer, }
- fetch (Fetch a record by ID)
inputs: { id: string, }
- create (Create a new record)
inputs: { title: string, content: string, }
If this fails, check:
- Is your server command correct? (Try running it manually)
- Does your server implement the MCP
initializehandshake? - Are tools properly advertised with
inputSchema?
Step 4: Test Individual Tool Calls
Before recording, test each tool manually:
# Test the search tool
# Test the fetch tool
This verifies:
- Tools are callable
- Arguments are accepted
- Responses are valid JSON
- No runtime errors
Step 5: Record Baseline Snapshot
Once individual tools work, record the baseline:
# Interactively record exchanges
The tool will:
- Connect to your server
- List all tools
- Prompt you to select tools to test
- Ask for input arguments for each
- Record the request and response
- Save everything to
baseline.trip-test.json
Example session:
Connected to: python -m my_mcp_server
Tools available:
1. search (Search records)
2. fetch (Fetch a record by ID)
Enter tool number to record (or 0 to finish): 1
Enter arguments as JSON: {"q":"test","limit":5}
Calling search...
Result: {"results":[...],"total":42}
✓ Exchange recorded!
Enter tool number to record (or 0 to finish): 2
Enter arguments as JSON: {"id":"123"}
Calling fetch...
Result: {"id":"123","title":"...","content":"..."}
✓ Exchange recorded!
Enter tool number to record (or 0 to finish): 0
Snapshot saved to: contracts/baseline.trip-test.json
Step 6: Inspect the Snapshot
View what was recorded:
# Pretty-print the snapshot
|
# Or use your editor
The snapshot captures:
- Tool schemas (names, descriptions, input parameters)
- Recorded exchanges (what you called, what you got back)
- Metadata (when recorded, server version)
Step 7: Commit Baseline
Lock in your contract:
Testing Workflow Scenarios
Scenario 1: Developer Makes Safe Change
# Developer adds an optional parameter (safe)
# Edits server code and rebuilds
# Run contract check
# Output:
# Replaying ex-001: search
# Input: {"q": "test", "limit": 5}
# ✓ PASS
#
# Results: 1 passed, 0 failed
CI will pass ✅
Scenario 2: Developer Makes Breaking Change
# Developer renames a required parameter
# Before: search(q, limit)
# After: search(q, max_results)
# Developer commits and opens PR
# CI automatically runs
# Output:
# 🚨 BREAKING CHANGES:
# • search: Parameter 'limit' removed (was required: true)
# • search: Parameter 'max_results' added (required: true)
#
# Overall Verdict: BREAKING
CI will fail ❌ and block the PR
Resolution options:
Option A: Fix the breaking change
# Revert to old parameter name
# CI passes on new commit
Option B: Intentionally break the contract
# Update baseline to reflect new contract
# Commit the new baseline
# New baseline is now locked in
# Future PRs will compare against this
Scenario 3: Detect Non-Breaking Change
# Developer improves tool descriptions
# Before: "Search records"
# After: "Search records by query (case-insensitive)"
# Output:
# ⚠️ NON-BREAKING CHANGES:
# • search: Description changed
#
# Overall Verdict: SAFE (no breaking changes)
CI will pass ✅
Real Server Examples
Python MCP Server (with venv)
# Create venv
# Test connection
# Config file
Node.js MCP Server
# Install dependencies
# Build if needed
# Test connection
# Config file
Rust MCP Server
# Build in release mode for performance
# Test connection
# Config file with proper invocation
Docker Container
# Build your MCP server image
# Test connection
# Config file
Testing Multiple Versions
Keep separate configs for different versions:
# test-v1.toml
# test-v2.toml
# Test both
Testing Against Example Servers
trip-test ships with example MCP servers for learning:
# Clone trip-test
# Build example servers
# Test against basic example
# Record baseline from example
# Test against modified example
# Expected output: BREAKING changes detected ❌
How It Works
Architecture
┌──────────────────┐
│ Your CLI │
│ (trip-test) │
└────────┬─────────┘
│ JSON-RPC (stdio)
│
┌────────▼─────────┐
│ Your MCP Server │
│ (any language) │
└──────────────────┘
- Connect: Spawn your MCP server as a subprocess
- Handshake: Perform MCP
initializeprotocol - List: Fetch all tools + input schemas
- Call: Invoke tools with your args
- Record: Save exchanges to snapshot JSON
- Diff: Compare baseline schema vs live schema
- Classify: Mark each change as breaking/non-breaking/additive
Change Classification Rules
| Change | Class | Example |
|---|---|---|
| Tool removed | BREAKING | search tool deleted |
| Tool added | ADDITIVE | New fetch tool added |
| Required param removed | BREAKING | limit param removed |
| Required param added | BREAKING | New required max_results |
| Optional param added | ADDITIVE | Optional sort param added |
| Type narrowed | BREAKING | string → enum |
| Type widened | NON-BREAKING | enum → string |
| Enum value removed | BREAKING | Removed option "asc" |
| Enum value added | NON-BREAKING | Added option "desc" |
| Description changed | NON-BREAKING | Help text updated |
Output Formats
Pretty (default)
Human-readable terminal output with colors.
JSON (for CI)
Machine-readable JSON:
Limitations & Known Issues
MVP Scope (current):
- Supports stdio transport only (spawn subprocess)
- Flat + one level of nested schema changes (handles most cases)
- No
$refresolution (document workaround) - No credential management (pass headers via
--headerflag planned) - No output schema validation (input schemas only)
Planned (Phase 2+):
- HTTP transport (remote servers)
- Web UI (interactive tool browser)
- Watch mode (alert on drift)
- Snapshot registry (share baselines)
- Test generation (auto-generate snapshots from schemas)
Contributing
We welcome contributions! See CONTRIBUTING.md for guidelines.
Local Development
# Clone repo
# Build
# Test against example server
# Run tests
# Lint
# Format
Project Structure
trip-test/
├── src/
│ ├── main.rs # CLI entry point
│ ├── lib.rs # Public API
│ ├── mcp.rs # MCP protocol client
│ ├── snapshot.rs # Snapshot file format
│ ├── diff.rs # Diff engine & classification
│ ├── config.rs # TOML configuration
│ └── logger.rs # Logging utilities
├── examples/
│ ├── test-server.rs # Dummy MCP server for testing
│ └── test-server-modified.rs
├── .github/workflows/
│ └── trip-test.yml # CI workflow
├── Cargo.toml
├── LICENSE
└── README.md
License
MIT License — see LICENSE for details.
Acknowledgments
Inspired by:
- Postman — API testing
- Jest snapshots — regression detection
- OpenAPI/Swagger — schema-first design
Support
- GitHub Issues: Report bugs or request features
- Discussions: Ask questions
- Documentation: Full docs
Roadmap
- Core MVP (snapshot + diff + CI)
- GitHub Actions integration
- JSON output for CI/CD
- Configuration file support
- Published on crates.io
- HTTP transport support
- Web UI dashboard
- Watch mode & alerts
- Snapshot registry
- Test generation
Made with ❤️ for MCP developers
Questions? Open an issue or discussion.