# mecha10-node-runner
Node lifecycle manager for Mecha10 - handles pre-flight checks, model management, and process supervision.
## Overview
`mecha10-node-runner` is a binary that acts as a wrapper around Mecha10 node binaries. It ensures all prerequisites are met before executing a node, providing production-ready supervision with:
- **Pre-flight Checks** - Validates environment (Redis, disk space, ports, permissions, GPU)
- **Model Management** - Automatic download of AI models from HuggingFace
- **Dependency Ordering** - Waits for required nodes to be ready before starting
- **Process Supervision** - Spawns and monitors node processes
- **Restart Policies** - Automatic recovery from failures with exponential backoff
- **Health Monitoring** - Periodic process health checks
- **Resource Limits** - Memory and CPU monitoring with warnings and enforcement
- **Config Loading** - Loads node configuration with automatic fallback chains
- **Binary Resolution** - Resolves node binary paths (monorepo vs standalone)
- **Environment Setup** - Sets up environment variables for consistent execution
- **Log Management** - Creates log files and redirects output
## Why?
Previously, model checking and pulling logic was embedded in the CLI's `mecha10 dev` command. This meant:
- Model pulling only worked in development mode
- Edge deployments couldn't automatically pull models
- Code duplication between CLI and launcher
- Poor separation of concerns
- No consistent runtime supervision
With `mecha10-node-runner`, lifecycle logic is centralized and works consistently across:
- **Development** - via `mecha10 dev`
- **Edge** - via `mecha10-launcher`
- **Standalone** - direct execution
## Usage
### Direct execution
```bash
# Run a node directly
mecha10-node-runner camera
# With custom project name
mecha10-node-runner --project rover-robot camera
# With restart policy (on-failure with 5 retries)
mecha10-node-runner --restart-policy on-failure --max-retries 5 camera
# With restart policy (always restart with custom backoff)
mecha10-node-runner --restart-policy always --backoff 2 camera
# With health monitoring (check every 30 seconds)
mecha10-node-runner --health-check-interval 30 camera
# Combine restart policy and health monitoring
mecha10-node-runner --restart-policy on-failure --max-retries 3 --health-check-interval 30 camera
```
### Via CLI (development)
```bash
# CLI spawns node-runner internally
mecha10 dev
```
### Via Launcher (edge)
```bash
# Launcher spawns node-runner internally
mecha10-launcher start camera
```
## How it works
When you run `mecha10-node-runner camera`:
1. **Load config** - Reads `configs/*/nodes/camera/config.json`
2. **Check models** - If config specifies a model, checks if it exists
3. **Pull models** - Downloads from HuggingFace if missing
4. **Resolve binary** - Finds `target/release/camera` or `target/release/{project-name}`
5. **Setup environment** - Sets `NODE_NAME`, `REDIS_URL`, `RUST_LOG`, etc.
6. **Setup logs** - Creates `logs/camera.log`
7. **Supervise execution** - Spawns and monitors the node process with restart/health policies
8. **Monitor health** - Periodically checks if process is still running
9. **Handle failures** - Restarts node according to policy with exponential backoff
## Architecture
```
┌─────────────────────────────────────────────────────────┐
│ Development Tools (mecha10 CLI) │
│ Fleet Management (mecha10-launcher) │
└────────────────┬────────────────────────────────────────┘
│ spawns
┌────────────────▼────────────────────────────────────────┐
│ mecha10-node-runner (this package) │
│ • Pre-flight checks │
│ • Model pulling │
│ • Config loading │
└────────────────┬────────────────────────────────────────┘
│ execs
┌────────────────▼────────────────────────────────────────┐
│ Node Binaries (camera, lidar, etc.) │
│ • Pure business logic │
│ • Assumes models and config are ready │
└─────────────────────────────────────────────────────────┘
```
## Dependencies
- **mecha10-cli** - For ModelService (model catalog and downloading)
- **mecha10-core** - For Context (config loading)
- **indicatif** - Progress bars for model downloads
- **clap** - CLI argument parsing
- **tokio** - Async runtime
## Restart Policies
The node-runner supports three restart policies:
### 1. Never (default)
Never restart on failure. Node exits on first failure.
```bash
mecha10-node-runner --restart-policy never camera
```
### 2. On-Failure
Restart only when node exits with non-zero status. Uses exponential backoff (1s, 2s, 4s, 8s...).
```bash
mecha10-node-runner --restart-policy on-failure --max-retries 3 --backoff 1 camera
```
Parameters:
- `--max-retries`: Maximum restart attempts (default: 3)
- `--backoff`: Initial backoff duration in seconds (default: 1s)
Example backoff sequence:
- Attempt 1: 1s delay
- Attempt 2: 2s delay
- Attempt 3: 4s delay
- Attempt 4: Fails permanently
### 3. Always
Restart indefinitely regardless of exit status. Uses constant backoff.
```bash
mecha10-node-runner --restart-policy always --backoff 2 camera
```
Parameters:
- `--backoff`: Backoff duration in seconds between restarts (default: 1s)
## Health Monitoring
Health monitoring periodically checks if the node process is still running:
```bash
mecha10-node-runner --health-check-interval 30 camera
```
Parameters:
- `--health-check-interval`: Health check interval in seconds (default: 30s, 0 to disable)
Implementation:
- **Unix**: Uses `kill(pid, SIGCONT)` to check process existence
- **Windows**: Uses `sysinfo` to query process status
Health status changes are logged:
- `Running` → Process is alive
- `Stopped` → Process has exited (triggers restart if policy allows)
## Configuration File
Node configuration can specify runtime settings in `config.json`:
```json
{
"runtime": {
"restart_policy": "on-failure",
"max_retries": 5,
"backoff_secs": 2,
"memory_limit_mb": 512,
"cpu_limit_percent": 50.0,
"health_check_interval_secs": 30
},
"depends_on": ["redis", "database"],
"startup_timeout_secs": 30,
"preflight": {
"check_ports": [8080],
"min_disk_space_gb": 1,
"require_gpu": false
},
"models": {
"vision_model": {
"name": "yolov8n",
"source": "huggingface",
"repo": "Ultralytics/YOLOv8",
"file": "yolov8n.onnx"
}
}
}
```
CLI arguments override config file settings.
## Pre-flight Checks
The node-runner performs comprehensive environment validation before starting nodes:
### Redis Connectivity
Verifies Redis is reachable and responsive:
```bash
# Checks connection and performs PING
mecha10-node-runner camera
```
**Output:**
```
🔍 Running pre-flight checks...
Checking Redis connectivity at redis://localhost:6379...
✓ Redis is reachable
```
**Error Example:**
```
❌ Pre-flight check failed: Redis not reachable at redis://localhost:6379
Possible solutions:
1. Start Redis: docker compose up -d redis
2. Check REDIS_URL environment variable
3. Skip this check with: --skip-preflight (not recommended)
```
### Disk Space
Verifies sufficient disk space is available:
```bash
# Default: 1 GB minimum
mecha10-node-runner camera
```
**Output:**
```
Checking disk space (minimum: 1.0 GB)...
✓ Available: 128.5 GB
```
**Error Example:**
```
❌ Pre-flight check failed: Insufficient disk space (0.5 GB < 1.0 GB required)
Possible solutions:
1. Free up disk space
2. Reduce min_disk_space_gb requirement in config
```
### Port Availability
Checks that required ports are not in use:
```json
{
"preflight": {
"check_ports": [8080, 9090]
}
}
```
**Output:**
```
Checking port 8080 is available...
✓ Port 8080 is available
Checking port 9090 is available...
✓ Port 9090 is available
```
**Error Example:**
```
❌ Pre-flight check failed: Port 8080 is already in use
Possible solutions:
1. Stop the process using port 8080
2. Use a different port in your config
```
### File Permissions
Verifies write access to required directories:
```bash
# Checks logs/ directory by default
mecha10-node-runner camera
```
**Output:**
```
Checking write access to logs/...
✓ Can write to logs/
```
**Error Example:**
```
❌ Pre-flight check failed: Cannot write to logs/
Possible solutions:
1. Create directory: mkdir -p logs
2. Fix permissions: chmod 755 logs
```
### GPU Detection
Optionally checks for GPU availability:
```json
{
"preflight": {
"require_gpu": true
}
}
```
**Output (GPU found):**
```
Checking GPU availability...
✓ Found NVIDIA GPU: NVIDIA GeForce RTX 3080
```
**Output (no GPU):**
```
Checking GPU availability...
⚠ No GPU detected (proceeding with CPU)
```
**Error Example (GPU required but not found):**
```
❌ Pre-flight check failed: GPU required but not available
Possible solutions:
1. Install GPU drivers
2. Check CUDA installation
3. Set require_gpu: false in config for CPU-only execution
```
## Resource Limits
The node-runner monitors and enforces resource limits to prevent runaway processes:
### Memory Limits
Track memory usage and terminate if exceeded:
```json
{
"runtime": {
"memory_limit_mb": 512
}
}
```
**Behavior:**
- **80% threshold:** Warning logged
- **100% threshold:** Process terminated
**Example Logs:**
```
WARN Node 'camera' memory usage at 81.2% of limit (415 MB / 512 MB)
ERROR Node 'camera' exceeded memory limit (520 MB / 512 MB) - terminating
```
### CPU Limits
Track CPU usage and terminate if exceeded:
```json
{
"runtime": {
"cpu_limit_percent": 50.0
}
}
```
**Behavior:**
- **80% threshold:** Warning logged
- **100% threshold:** Process terminated
**Example Logs:**
```
WARN Node 'camera' CPU usage at 82.5% of limit (41.2% / 50.0%)
ERROR Node 'camera' exceeded CPU limit (52.3% / 50.0%) - terminating
```
### Resource Monitoring
Resource checks run during health check intervals:
```bash
# Check every 30 seconds
mecha10-node-runner --health-check-interval 30 camera
```
**Implementation:**
- Uses `sysinfo` crate for cross-platform monitoring
- Works on Linux, macOS, and Windows
- Minimal performance overhead
## Dependency Ordering
Ensure nodes start in correct order by waiting for dependencies:
```json
{
"depends_on": ["redis", "database"],
"startup_timeout_secs": 30
}
```
**How it works:**
1. Node-runner checks for each dependency
2. Polls Redis key `/_internal/node/{dep}/alive` every 500ms
3. Waits up to `startup_timeout_secs` for each dependency
4. Fails with clear error if timeout exceeded
**Example Output:**
```
Waiting for dependencies: redis, database
✓ redis is ready
✓ database is ready
Starting node...
```
**Error Example:**
```
❌ Dependency timeout: 'database' not ready after 30 seconds
Possible solutions:
1. Check that 'database' node is running
2. Increase startup_timeout_secs in config
3. Verify Redis connectivity
```
## Troubleshooting
### Pre-flight Check Failures
**Redis not reachable:**
```bash
# Check Redis status
docker compose ps redis
# Start Redis
docker compose up -d redis
# Test connection
redis-cli ping
```
**Insufficient disk space:**
```bash
# Check available space
df -h
# Clean up old logs
rm logs/*.log.old
# Or reduce requirement in config
```
**Port already in use:**
```bash
# Find process using port
lsof -i :8080 # Linux/Mac
# Kill process or use different port
```
**Permission denied:**
```bash
# Create logs directory
mkdir -p logs
# Fix permissions
chmod 755 logs
# Or run with appropriate user
```
**GPU not found:**
```bash
# Check GPU drivers (NVIDIA)
nvidia-smi
# Check CUDA installation
nvcc --version
# Or set require_gpu: false for CPU-only
```
### Restart Loop Debugging
If a node is restarting repeatedly:
**1. Check node logs:**
```bash
tail -f logs/camera.log
```
**2. Disable restart temporarily:**
```json
{
"runtime": {
"restart_policy": "never"
}
}
```
**3. Run node directly:**
```bash
# Run node without supervisor to see errors
target/release/camera
```
**4. Common causes:**
- Configuration error (fix config.json)
- Missing hardware (check device is connected)
- Permission denied (run with appropriate privileges)
- Resource exhaustion (increase limits or reduce load)
### Resource Limit Issues
**Node killed for exceeding memory:**
**Check current usage:**
```bash
# Monitor node memory
**Solutions:**
1. Increase limit if usage is legitimate
2. Investigate memory leak if unexpected
3. Optimize node code to use fewer resources
**Node killed for exceeding CPU:**
**Check current usage:**
```bash
# Monitor node CPU
top -p $(pgrep camera)
```
**Solutions:**
1. Increase limit if usage is legitimate
2. Profile node to find CPU hotspots
3. Optimize expensive operations
### Dependency Timeout
**Database dependency not ready after 30s:**
**Check dependency status:**
```bash
# Is the node running?
# Check logs
tail -f logs/database.log
```
**Solutions:**
1. Increase `startup_timeout_secs` in config
2. Fix dependency node startup issues
3. Verify Redis connectivity for readiness checks
### Model Download Failures
**Model download fails from HuggingFace:**
**Check internet connection:**
```bash
curl -I https://huggingface.co
```
**Verify model paths:**
```json
{
"models": {
"vision_model": {
"repo": "Ultralytics/YOLOv8", # Check this exists
"file": "yolov8n.onnx" # Check filename is correct
}
}
}
```
**Solutions:**
1. Check internet connectivity
2. Verify HuggingFace repo/file paths
3. Download manually and place in models/ directory
## Future enhancements
- Health checks before start (Redis, GPU, network)
- Resource limits (CPU, memory)
- Telemetry hooks
- Dependency checking (ensure other nodes are running)
- GPU allocation
## Remote Node Control (Dashboard/API)
When running in development mode (`mecha10 dev`), nodes can be controlled remotely via the dashboard or API:
### Stop via Dashboard
When you click "Stop" on a node in the dashboard:
1. Dashboard publishes a `stop` command to Redis (`system/cli/command`)
2. CLI's CommandListener receives the command
3. CLI sends SIGTERM to the node-runner process group
4. Node-runner's supervisor receives SIGTERM and forwards it to the child node
5. Both node-runner and child process exit cleanly
6. **No restart occurs** - SIGTERM signals intentional shutdown
**Important:** Even if a node has `restart_policy: "always"`, stopping via dashboard will NOT trigger a restart. The supervisor only restarts nodes that exit unexpectedly, not those terminated via signal.
### Start via Dashboard
When you click "Start" on a stopped node:
1. Dashboard publishes a `start` command to Redis
2. CLI's CommandListener receives the command
3. CLI spawns a new node-runner process (or direct node process in standalone mode)
4. Node starts with configured restart policy
### Restart via Dashboard
When you click "Restart":
1. Stop command is sent (as above)
2. Brief delay for cleanup (500ms)
3. Start command is sent (as above)
### When Restart Policies DO Apply
Restart policies trigger when:
- Node process crashes (non-zero exit code)
- Node process terminates unexpectedly
- Parent node-runner is still running
Restart policies do NOT trigger when:
- CLI sends SIGTERM (stop command from dashboard)
- User presses Ctrl+C in terminal
- `mecha10 dev` session ends
- Node-runner itself is terminated
### Troubleshooting Unexpected Restarts
If a node seems to restart after being stopped:
**1. Check if the correct process was stopped:**
```bash
# View running processes
# The node-runner should NOT be running after stop
```
**2. Check restart policy:**
```json
// configs/dev/nodes/<node>/config.json
{
"runtime": {
"restart_policy": "never" // Should be "never" to prevent restarts
}
}
```
**3. Check CLI logs:**
```bash
# The system log panel in dev mode shows:
# 🛑 [Dashboard] Stopping node 'speaker'...
# ✅ [Dashboard] Stopped node 'speaker'
```
**4. Check command-listener debug log:**
```bash
cat /tmp/command-listener.log
```
## Related
- [mecha10-runtime](../runtime/README.md) - In-process node supervision library
- [mecha10-cli](../cli/README.md) - Development CLI
- [mecha10-launcher](../launcher-service/README.md) - Fleet management service