# mecha10-video
WebRTC-based low-latency video streaming for robotics applications.
## Features
- **WebRTC Streaming**: H.264 video encoding with OpenH264
- **Multi-client Support**: Broadcast frames to multiple WebRTC connections
- **Low Latency**: 30-50ms glass-to-glass latency
- **Zero-copy Frame Sharing**: Efficient frame distribution using Arc
- **Optional Diagnostics**: Enable with `diagnostics` feature flag
## Architecture
```text
┌─────────────┐
│ Source │ (Godot, Camera, etc.)
│ (Frames) │
└──────┬──────┘
│ mpsc::channel
▼
┌─────────────────────┐
│ WebRTCServer │
│ (Frame Broadcast) │
└──────┬──────────────┘
│ broadcast::channel
├─────────┬─────────┐
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────┐
│ Conn 1 │ │ Conn 2 │ │ Conn 3 │
│ (H.264)│ │ (H.264)│ │ (H.264)│
└────┬───┘ └────┬───┘ └────┬───┘
│ │ │
▼ ▼ ▼
Browser Browser Browser
```
## Usage
### Basic Example
```rust
use mecha10_video::{CameraFrame, ImageFormat, WebRTCServer};
use mecha10_video::signaling::start_signaling_server;
use tokio::sync::mpsc;
use std::sync::Arc;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Create frame channel
let (frame_tx, frame_rx) = mpsc::channel(10);
// Create WebRTC server (without diagnostics)
let webrtc_server = WebRTCServer::new(frame_rx).await?;
// Start signaling server for SDP exchange
let server = Arc::new(webrtc_server);
tokio::spawn(async move {
start_signaling_server(11010, server).await.unwrap();
});
// Send frames to WebRTC server
let frame = CameraFrame {
camera_id: "camera0".to_string(),
width: 640,
height: 480,
timestamp: 0,
image_bytes: Arc::new(vec![0u8; 640 * 480 * 3]),
format: ImageFormat::Rgb,
};
frame_tx.send(frame).await?;
Ok(())
}
```
### With Diagnostics
Enable the `diagnostics` feature in your `Cargo.toml`:
```toml
[dependencies]
mecha10-video = { path = "../video", features = ["diagnostics"] }
```
Then create the server with diagnostics:
```rust
use mecha10_diagnostics::prelude::StreamingCollector;
use std::sync::Arc;
let diagnostics = Arc::new(StreamingCollector::new("my-node"));
let webrtc_server = WebRTCServer::new(frame_rx, diagnostics).await?;
```
### Dual-Path Publishing (CameraPublisher)
For nodes that need to publish frames to multiple destinations (e.g., WebRTC for dashboards + Redis for telemetry), use `CameraPublisher`:
```rust
use mecha10_video::{CameraPublisher, ImageFormat, WebRTCServer};
use tokio::sync::mpsc;
use std::sync::Arc;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Create channels
let (webrtc_tx, webrtc_rx) = mpsc::channel(1);
let (redis_tx, redis_rx) = mpsc::channel(50);
// Create WebRTC server
let diagnostics = Arc::new(StreamingCollector::new("my-node"));
let webrtc_server = WebRTCServer::new(webrtc_rx, diagnostics.clone()).await?;
// Create camera publisher for dual-path streaming
let camera_publisher = CameraPublisher::new(webrtc_tx, diagnostics)
.with_secondary_publisher(redis_tx);
// Publish frame (zero-copy Arc sharing between WebRTC and Redis!)
let stats = camera_publisher.publish(
"camera0".to_string(),
vec![0u8; 640 * 480 * 3],
ImageFormat::Rgb,
640,
480,
12345,
)?;
println!("Published to {} destinations", stats.destinations);
Ok(())
}
```
**Key Features:**
- **Zero-copy sharing**: Wraps frame data in `Arc` once, shares between all destinations
- **Best-effort secondary**: Redis/telemetry publishing is non-blocking (drops on full channel)
- **Automatic diagnostics**: Tracks frame sent/dropped metrics (when diagnostics feature enabled)
- **Simple API**: Just call `publish()` with raw frame data
## Performance
- **Encoding**: ~8-15ms per frame (160×120 @ 30 FPS)
- **Latency**: 30-50ms glass-to-glass
- **Bitrate**: 400 Kbps (configurable in source)
- **Codec**: H.264 (OpenH264)
- **Resolution**: Supports any resolution (tested with 160×120, 640×480)
- **FPS**: Configurable (tested with 20-30 FPS)
## API Documentation
### Frame Types
#### `CameraFrame`
Represents a single camera frame.
```rust
pub struct CameraFrame {
pub camera_id: String,
pub width: u32,
pub height: u32,
pub timestamp: u64,
pub image_bytes: Arc<Vec<u8>>,
pub format: ImageFormat,
}
```
#### `ImageFormat`
Image data format.
```rust
pub enum ImageFormat {
Jpeg, // JPEG-compressed image
Rgb, // Raw RGB pixels (width × height × 3 bytes)
}
```
#### `FrameBroadcaster`
Efficiently broadcasts frames to multiple subscribers.
```rust
let broadcaster = FrameBroadcaster::new(1); // capacity = 1 for minimal latency
broadcaster.broadcast(frame)?;
let mut rx = broadcaster.subscribe();
```
### WebRTC Server
#### `WebRTCServer`
Creates and manages WebRTC connections.
```rust
// Without diagnostics
let server = WebRTCServer::new(frame_rx).await?;
// With diagnostics (requires "diagnostics" feature)
let server = WebRTCServer::new(frame_rx, diagnostics).await?;
// Create connection for a client
let connection = server.create_connection().await?;
```
#### `WebRTCConnection`
Represents a single client connection.
```rust
// Create SDP offer
let offer_sdp = connection.create_offer().await?;
// Handle SDP answer from browser
connection.handle_answer(answer_sdp).await?;
// Add ICE candidate
connection.add_ice_candidate(candidate, sdp_mid, sdp_mline_index).await?;
// Start streaming (blocks until connection closes)
connection.run_streaming_loop().await?;
```
### Signaling Server
#### `start_signaling_server`
Starts a WebSocket-based WebRTC signaling server.
```rust
use mecha10_video::start_signaling_server;
let webrtc_server = Arc::new(webrtc_server);
tokio::spawn(async move {
start_signaling_server(11010, webrtc_server).await.unwrap();
});
```
Protocol:
- Endpoint: `ws://localhost:11010/webrtc`
- Single-connection mode (disconnects previous client when new one connects)
- Server-initiated SDP offer flow
## Integration
### Simulation Bridge
The `simulation-bridge` node uses this package for camera streaming:
```rust
use mecha10_video::{CameraFrame, WebRTCServer, start_signaling_server};
// Create WebRTC server with diagnostics
let webrtc_server = WebRTCServer::new(frame_rx, diagnostics).await?;
// Start signaling server
tokio::spawn(async move {
start_signaling_server(11010, Arc::new(webrtc_server)).await
});
```
### Edge Robot Cameras
For physical cameras on edge robots:
1. Capture frames from camera driver
2. Convert to `CameraFrame` with RGB or JPEG format
3. Send to WebRTC server via mpsc channel
4. Dashboard connects via WebRTC signaling server
## Source of Truth
This package was extracted from `simulation-bridge/src/webrtc_server.rs` and `simulation-bridge/src/signaling_server.rs`, which are the production-tested reference implementations.
See [`docs/VIDEO_EXTRACTION_PLAN.md`](../../docs/VIDEO_EXTRACTION_PLAN.md) for extraction history and rationale.
## Dependencies
- `webrtc 0.14` - WebRTC implementation
- `openh264 0.9` - Fast H.264 encoder
- `axum 0.7` - WebSocket signaling server
- `tokio` - Async runtime
- `image 0.25` - JPEG decoding (minimal features)
- `mecha10-diagnostics` (optional) - Streaming metrics
## License
MIT