# mecha10-diagnostics
Topic-based diagnostics and performance monitoring service for Mecha10.
## Overview
The diagnostics service provides comprehensive performance monitoring through the framework's pub/sub system. All diagnostics are published to topics under the `/diagnostics` namespace, allowing real-time monitoring via CLI, dashboard, or custom subscribers.
## Features
- **Topic-based architecture**: Seamlessly integrates with Mecha10's pub/sub system
- **Streaming pipeline metrics**: Frame pipeline, latency, encoding performance, bandwidth
- **WebRTC metrics**: Connection stats, quality metrics (RTT, packet loss, jitter)
- **WebSocket metrics**: Connection tracking, message rates
- **Redis metrics**: Connection pool, operation latency and throughput
- **Docker metrics**: Container CPU, memory, network, and I/O stats
- **System metrics**: Host CPU, memory, disk, and network usage
- **Low overhead**: Atomic counters on hot paths, background aggregation
## Diagnostic Topics
All diagnostics use the `/diagnostics` namespace:
```
/diagnostics/streaming/pipeline # Frame pipeline metrics
/diagnostics/streaming/latency # Latency measurements
/diagnostics/streaming/encoding # Encoding performance
/diagnostics/streaming/bandwidth # Bandwidth usage
/diagnostics/webrtc/connections # WebRTC peer connections
/diagnostics/webrtc/quality # RTT, packet loss, jitter
/diagnostics/websocket/connections # WebSocket connections
/diagnostics/websocket/messages # WebSocket message stats
/diagnostics/redis/pool # Redis connection pool
/diagnostics/redis/operations # Redis operation metrics
/diagnostics/docker/containers # Docker container stats
/diagnostics/godot/performance # Godot FPS, frame time, physics
/diagnostics/godot/scene # Godot node count, memory
/diagnostics/godot/connection # Godot WebSocket health
/diagnostics/system/resources # System-wide resources
/diagnostics/node/{id}/health # Per-node health
```
## Usage
### Publishing Streaming Diagnostics
```rust
use mecha10_diagnostics::prelude::*;
use mecha10_core::prelude::*;
// Create collector
let collector = StreamingCollector::new("mujoco-sim");
// Record metrics on hot path (minimal overhead)
collector.record_frame_received();
collector.record_frame_encoded(5000); // 5ms encode time
collector.record_frame_sent(10000); // 10KB frame
// Publish aggregated metrics periodically (every 1-5 seconds)
collector.publish_all(&ctx, 400_000).await?;
```
### Subscribing to Diagnostics
```rust
use mecha10_diagnostics::prelude::*;
use mecha10_core::prelude::*;
// Subscribe to streaming pipeline metrics
let mut rx = ctx.subscribe(
Topic::<DiagnosticMessage<StreamingPipelineMetrics>>::new(
TOPIC_DIAGNOSTICS_STREAMING_PIPELINE
)
).await?;
while let Some(msg) = rx.recv().await {
println!("FPS: {}, Frames dropped: {}",
msg.payload.fps,
msg.payload.frames_dropped
);
}
```
### Docker Metrics
```rust
// Create Docker collector
let docker = DockerCollector::new("diagnostics-service").await;
// Collect metrics for all containers
docker.collect_all_containers(&ctx).await?;
```
### System Metrics
```rust
// Create system collector
let system = SystemCollector::new("diagnostics-service");
// Collect and publish system metrics
system.collect_metrics(&ctx).await?;
```
### Godot Metrics
```rust
// Create Godot collector
let godot = GodotCollector::new(
"simulation-bridge",
"ws://localhost:11008".to_string(), // Control URL
"ws://localhost:11009".to_string(), // Camera URL
);
// Track connection events
godot.set_control_connected(true);
godot.set_camera_connected(true);
// Record messages
godot.record_control_message(now_micros());
godot.record_camera_frame(now_micros());
// Publish connection health
godot.publish_connection_metrics(&ctx).await?;
// Publish performance metrics (data from Godot)
godot.publish_performance_metrics(
&ctx,
60.0, // fps
60.0, // target_fps
16.6, // frame_time_ms
2.5, // physics_time_ms
10.0, // render_time_ms
4.1, // idle_time_ms
).await?;
```
## CLI Integration
Monitor diagnostics in real-time:
```bash
# View all diagnostics
mecha10 diagnostics
# Filter by category
mecha10 diagnostics --category streaming
mecha10 diagnostics --category docker
# Live streaming metrics only
mecha10 diagnostics --streaming
# Historical query
mecha10 diagnostics --from "2024-01-01 12:00" --to "2024-01-01 13:00"
# Export to file
mecha10 diagnostics --export metrics.json
```
## Dashboard Integration
The diagnostics service integrates with the dashboard via WebSocket:
- Real-time charts and visualizations
- Bottleneck detection
- Alerting on anomalies
- Historical analysis
Access at: `http://localhost:3000/dashboard/diagnostics`
## Metric Types
### Counter
Monotonically increasing value (e.g., frames received)
```rust
let counter = Counter::new();
counter.inc();
counter.add(5);
let total = counter.get();
```
### Gauge
Current value that can go up or down (e.g., queue depth)
```rust
let gauge = Gauge::new();
gauge.set(10);
gauge.inc();
gauge.dec();
```
### Histogram
Distribution tracking with percentiles (e.g., encoding latency)
```rust
let mut hist = Histogram::for_latency()?;
hist.record(5000); // Record 5ms
let p95 = hist.p95();
let p99 = hist.p99();
```
## Performance
The diagnostics system is designed for minimal overhead:
- **Atomic counters**: Zero-cost increment on hot paths
- **Background aggregation**: Expensive operations (percentiles) run off hot path
- **Rate limiting**: Automatic 1-second intervals for publishing
- **Zero-copy**: Efficient Arc-based data sharing
Overhead on streaming pipeline: **< 1% CPU**, **< 1MB memory**
## Architecture
```
┌─────────────────────────────────────────┐
│ Diagnostic Publishers │
│ (Simulation Bridge, Nodes, Services) │
└────────────────┬────────────────────────┘
│
▼
┌───────────────┐
│ Redis Pub/Sub │
└───────────────┘
│
┌────────┴────────┐
▼ ▼
┌────────┐ ┌──────────┐
│ CLI │ │Dashboard │
└────────┘ └──────────┘
│ │
▼ ▼
┌──────────────────────────────┐
│ Telemetry Service │
│ (Historical Archive) │
└──────────────────────────────┘
```
## Future Enhancements
1. **Redis instrumentation**: Add detailed operation tracking to mecha10-core::Context
2. **WebRTC stats**: Extract metrics from WebRTC peer connections
3. **WebSocket tracking**: Instrument WebSocket connections
4. **Custom metrics**: Allow nodes to publish custom diagnostic metrics
5. **Alerting**: Automatic alerts on threshold violations
6. **Profiling integration**: Integration with performance profiling tools
## License
MIT