assay-core 2.9.0

High-performance evaluation framework for LLM agents (Core)
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150

# Assay

[![Crates.io](https://img.shields.io/crates/v/assay-cli.svg)](https://crates.io/crates/assay-cli)
[![CI](https://github.com/Rul1an/assay/actions/workflows/ci.yml/badge.svg)](https://github.com/Rul1an/assay/actions/workflows/ci.yml)
[![License](https://img.shields.io/crates/l/assay-core.svg)](https://github.com/Rul1an/assay/blob/main/LICENSE)

**Policy-as-Code for AI Agents.**
Deterministic testing, runtime enforcement, and verifiable evidence for the Model Context Protocol.

## Install

```bash
curl -fsSL https://getassay.dev/install.sh | sh
```

Or via Cargo:
```bash
cargo install assay-cli
```

## Core Workflow

### 1. Record → Replay → Validate

Record agent behavior once, replay deterministically in CI. No LLM calls, no flakiness.

```bash
# Capture traces from your agent
assay import --format mcp-inspector session.json --out trace.jsonl

# Validate against policy (milliseconds, $0 cost)
assay validate --config assay.yaml --trace-file trace.jsonl

# CI gate with SARIF output
assay run --config assay.yaml --format sarif
```

### 2. Generate Policies from Behavior

```bash
# Single trace → policy
assay generate -i trace.jsonl --heuristics

# Multi-run profiling for stable policies
assay profile init --output profile.yaml --name my-app
assay profile update --profile profile.yaml -i trace.jsonl --run-id ci-123
assay generate --profile profile.yaml --min-stability 0.8
```

### 3. Evidence Bundles (Audit/Compliance)

Tamper-evident bundles with content-addressed IDs. CloudEvents v1.0 format.

```bash
# Export evidence
assay evidence export --profile profile.yaml --out bundle.tar.gz

# Verify integrity
assay evidence verify bundle.tar.gz

# Lint for security issues (SARIF)
assay evidence lint bundle.tar.gz --format sarif

# Compare runs
assay evidence diff baseline.tar.gz current.tar.gz
```

## Runtime Enforcement

### MCP Server Proxy

```bash
# Start policy enforcement proxy
assay mcp-server --policy policy.yaml
```

### Kernel-Level Sandbox (Linux)

```bash
# Landlock isolation (rootless)
assay sandbox --policy policy.yaml -- python agent.py

# eBPF/LSM enforcement (requires capabilities)
sudo assay monitor --policy policy.yaml --pid <agent-pid>
```

## Configuration

`assay.yaml`:
```yaml
version: "2.0"
name: "mcp-default-gate"

allow: ["*"]

deny:
  - "exec*"
  - "shell*"

constraints:
  - tool: "read_file"
    params:
      path:
        matches: "^/app/.*|^/data/.*"
```

## Python SDK

```bash
pip install assay
```

```python
from assay import AssayClient, validate

# Record traces
client = AssayClient("traces.jsonl")
client.record_trace(tool_call)

# Validate
result = validate("policy.yaml", traces)
assert result["passed"]
```

Pytest plugin for automatic trace capture:
```python
@pytest.mark.assay(trace_file="test_traces.jsonl")
def test_agent():
    pass
```

## Documentation

- [Getting Started](https://getassay.dev/docs/quickstart)
- [Policy Reference](docs/reference/policies.md)
- [Evidence Contract](docs/architecture/ADR-006-Evidence-Contract.md)
- [Runtime Architecture](docs/architecture/runtime.md)
- [Python SDK](docs/python-sdk/)

## Contributing

```bash
cargo test --workspace
```

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

[MIT](LICENSE)