union_square 0.2.0

A proxy/wire-tap service for making LLM calls and recording everything that happens in a session for later analysis and test-case extraction
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

# LLM Request Data Parsing Implementation Summary

## Problem Statement
The audit commands implementation was using placeholder data ("Placeholder prompt", "placeholder-model") instead of parsing actual LLM request data from HTTP request bodies. This violated event sourcing principles where events must record what actually happened.

## Solution Overview
Implemented a comprehensive solution to parse real LLM request data from various provider formats and integrate it into the event sourcing system.

## Components Implemented

### 1. LLM Request Parser (`llm_request_parser.rs`)
- Parses LLM requests from multiple provider formats:
  - **OpenAI format** (GPT models) - `/v1/chat/completions`, `/v1/completions`
  - **Anthropic format** (Claude models) - `/v1/messages`
  - **AWS Bedrock format** - `/bedrock/*`, `/invoke`
- Extracts common elements: model version, prompt, and parameters
- Provides graceful fallback for unknown formats
- Includes comprehensive error handling

### 2. Audit Buffer Manager (`audit_buffer.rs`)
- Manages buffering of request/response body chunks
- Reconstructs complete bodies from multiple chunks
- Handles out-of-order chunk delivery
- Provides cleanup functionality for completed requests

### 3. Enhanced Audit Commands (`audit_commands.rs`)
- Updated `RecordRequestReceived` to accept parsed LLM data
- Added `ProcessRequestBody` command for handling complete request bodies
- Integrated parser to extract real data instead of placeholders
- Maintains backward compatibility with existing event flow

### 4. Integration Guide (`audit_integration.md`)
- Comprehensive documentation for integrating the new functionality
- Example code for audit path handlers
- Two integration approaches: buffered and immediate

## Key Features

### Type Safety
- All parsing operations use validated domain types
- No primitive obsession - proper newtypes throughout
- Compile-time guarantees via Rust's type system

### Error Handling
- Graceful fallback for unknown request formats
- Detailed error messages for debugging
- System continues operating even with parsing failures

### Event Sourcing Compliance
- Events now record actual data, not placeholders
- Maintains immutability of events
- Preserves full audit trail of what happened

### Extensibility
- Easy to add new provider formats
- Parser auto-detects format based on URI and content
- Modular design allows independent testing

## Testing
- Comprehensive unit tests for all components
- Integration tests for command processing
- Property-based tests for buffer management
- All tests passing (361 passed, 0 failed)

## Usage Example

```rust
// When processing audit events
match &event.event_type {
    AuditEventType::RequestBody { content, .. } => {
        let command = ProcessRequestBody {
            // ... metadata fields
            body: content.clone(),
            timestamp: Timestamp::now(),
        };

        // This will parse the body and emit proper LlmRequestReceived event
        event_store.execute_command(command).await?;
    }
}
```

## Future Improvements
- Response body parsing (currently marked as TODO)
- Support for more LLM provider formats
- Streaming request body parsing
- Integration with metrics for model version tracking