mecha10-node-runner-0.8.0 is not a library.

mecha10-node-runner

Node lifecycle manager for Mecha10 - handles pre-flight checks, model management, and process supervision.

Overview

mecha10-node-runner is a binary that acts as a wrapper around Mecha10 node binaries. It ensures all prerequisites are met before executing a node, providing production-ready supervision with:

Pre-flight Checks - Validates environment (Redis, disk space, ports, permissions, GPU)
Model Management - Automatic download of AI models from HuggingFace
Dependency Ordering - Waits for required nodes to be ready before starting
Process Supervision - Spawns and monitors node processes
Restart Policies - Automatic recovery from failures with exponential backoff
Health Monitoring - Periodic process health checks
Resource Limits - Memory and CPU monitoring with warnings and enforcement
Config Loading - Loads node configuration with automatic fallback chains
Binary Resolution - Resolves node binary paths (monorepo vs standalone)
Environment Setup - Sets up environment variables for consistent execution
Log Management - Creates log files and redirects output

Why?

Previously, model checking and pulling logic was embedded in the CLI's mecha10 dev command. This meant:

Model pulling only worked in development mode
Edge deployments couldn't automatically pull models
Code duplication between CLI and launcher
Poor separation of concerns
No consistent runtime supervision

With mecha10-node-runner, lifecycle logic is centralized and works consistently across:

Development - via mecha10 dev
Edge - via mecha10-launcher
Standalone - direct execution

Usage

Direct execution

# Run a node directly
mecha10-node-runner camera

# With custom project name
mecha10-node-runner --project rover-robot camera

# With restart policy (on-failure with 5 retries)
mecha10-node-runner --restart-policy on-failure --max-retries 5 camera

# With restart policy (always restart with custom backoff)
mecha10-node-runner --restart-policy always --backoff 2 camera

# With health monitoring (check every 30 seconds)
mecha10-node-runner --health-check-interval 30 camera

# Combine restart policy and health monitoring
mecha10-node-runner --restart-policy on-failure --max-retries 3 --health-check-interval 30 camera

Via CLI (development)

# CLI spawns node-runner internally
mecha10 dev

Via Launcher (edge)

# Launcher spawns node-runner internally
mecha10-launcher start camera

How it works

When you run mecha10-node-runner camera:

Load config - Reads configs/*/nodes/camera/config.json
Check models - If config specifies a model, checks if it exists
Pull models - Downloads from HuggingFace if missing
Resolve binary - Finds target/release/camera or target/release/{project-name}
Setup environment - Sets NODE_NAME, REDIS_URL, RUST_LOG, etc.
Setup logs - Creates logs/camera.log
Supervise execution - Spawns and monitors the node process with restart/health policies
Monitor health - Periodically checks if process is still running
Handle failures - Restarts node according to policy with exponential backoff

Architecture

┌─────────────────────────────────────────────────────────┐
│           Development Tools (mecha10 CLI)                │
│           Fleet Management (mecha10-launcher)            │
└────────────────┬────────────────────────────────────────┘
                 │ spawns
┌────────────────▼────────────────────────────────────────┐
│           mecha10-node-runner (this package)             │
│  • Pre-flight checks                                     │
│  • Model pulling                                         │
│  • Config loading                                        │
└────────────────┬────────────────────────────────────────┘
                 │ execs
┌────────────────▼────────────────────────────────────────┐
│           Node Binaries (camera, lidar, etc.)            │
│  • Pure business logic                                   │
│  • Assumes models and config are ready                   │
└─────────────────────────────────────────────────────────┘

Dependencies

mecha10-cli - For ModelService (model catalog and downloading)
mecha10-core - For Context (config loading)
indicatif - Progress bars for model downloads
clap - CLI argument parsing
tokio - Async runtime

Restart Policies

The node-runner supports three restart policies:

1. Never (default)

Never restart on failure. Node exits on first failure.

mecha10-node-runner --restart-policy never camera

2. On-Failure

Restart only when node exits with non-zero status. Uses exponential backoff (1s, 2s, 4s, 8s...).

mecha10-node-runner --restart-policy on-failure --max-retries 3 --backoff 1 camera

Parameters:

--max-retries: Maximum restart attempts (default: 3)
--backoff: Initial backoff duration in seconds (default: 1s)

Example backoff sequence:

Attempt 1: 1s delay
Attempt 2: 2s delay
Attempt 3: 4s delay
Attempt 4: Fails permanently

3. Always

Restart indefinitely regardless of exit status. Uses constant backoff.

mecha10-node-runner --restart-policy always --backoff 2 camera

Parameters:

--backoff: Backoff duration in seconds between restarts (default: 1s)

Health Monitoring

Health monitoring periodically checks if the node process is still running:

mecha10-node-runner --health-check-interval 30 camera

Parameters:

--health-check-interval: Health check interval in seconds (default: 30s, 0 to disable)

Implementation:

Unix: Uses kill(pid, SIGCONT) to check process existence
Windows: Uses sysinfo to query process status

Health status changes are logged:

Running → Process is alive
Stopped → Process has exited (triggers restart if policy allows)

Configuration File

Node configuration can specify runtime settings in config.json:

{
  "runtime": {
    "restart_policy": "on-failure",
    "max_retries": 5,
    "backoff_secs": 2,
    "memory_limit_mb": 512,
    "cpu_limit_percent": 50.0,
    "health_check_interval_secs": 30
  },
  "depends_on": ["redis", "database"],
  "startup_timeout_secs": 30,
  "preflight": {
    "check_ports": [8080],
    "min_disk_space_gb": 1,
    "require_gpu": false
  },
  "models": {
    "vision_model": {
      "name": "yolov8n",
      "source": "huggingface",
      "repo": "Ultralytics/YOLOv8",
      "file": "yolov8n.onnx"
    }
  }
}

CLI arguments override config file settings.

Pre-flight Checks

The node-runner performs comprehensive environment validation before starting nodes:

Redis Connectivity

Verifies Redis is reachable and responsive:

# Checks connection and performs PING
mecha10-node-runner camera

Output:

🔍 Running pre-flight checks...
Checking Redis connectivity at redis://localhost:6379...
✓ Redis is reachable

Error Example:

❌ Pre-flight check failed: Redis not reachable at redis://localhost:6379

Possible solutions:
1. Start Redis: docker compose up -d redis
2. Check REDIS_URL environment variable
3. Skip this check with: --skip-preflight (not recommended)

Disk Space

Verifies sufficient disk space is available:

# Default: 1 GB minimum
mecha10-node-runner camera

Output:

Checking disk space (minimum: 1.0 GB)...
✓ Available: 128.5 GB

Error Example:

❌ Pre-flight check failed: Insufficient disk space (0.5 GB < 1.0 GB required)

Possible solutions:
1. Free up disk space
2. Reduce min_disk_space_gb requirement in config

Port Availability

Checks that required ports are not in use:

{
  "preflight": {
    "check_ports": [8080, 9090]
  }
}

Output:

Checking port 8080 is available...
✓ Port 8080 is available
Checking port 9090 is available...
✓ Port 9090 is available

Error Example:

❌ Pre-flight check failed: Port 8080 is already in use

Possible solutions:
1. Stop the process using port 8080
2. Use a different port in your config

File Permissions

Verifies write access to required directories:

# Checks logs/ directory by default
mecha10-node-runner camera

Output:

Checking write access to logs/...
✓ Can write to logs/

Error Example:

❌ Pre-flight check failed: Cannot write to logs/

Possible solutions:
1. Create directory: mkdir -p logs
2. Fix permissions: chmod 755 logs

GPU Detection

Optionally checks for GPU availability:

{
  "preflight": {
    "require_gpu": true
  }
}

Output (GPU found):

Checking GPU availability...
✓ Found NVIDIA GPU: NVIDIA GeForce RTX 3080

Output (no GPU):

Checking GPU availability...
⚠ No GPU detected (proceeding with CPU)

Error Example (GPU required but not found):

❌ Pre-flight check failed: GPU required but not available

Possible solutions:
1. Install GPU drivers
2. Check CUDA installation
3. Set require_gpu: false in config for CPU-only execution

Resource Limits

The node-runner monitors and enforces resource limits to prevent runaway processes:

Memory Limits

Track memory usage and terminate if exceeded:

{
  "runtime": {
    "memory_limit_mb": 512
  }
}

Behavior:

80% threshold: Warning logged
100% threshold: Process terminated

Example Logs:

WARN Node 'camera' memory usage at 81.2% of limit (415 MB / 512 MB)
ERROR Node 'camera' exceeded memory limit (520 MB / 512 MB) - terminating

CPU Limits

Track CPU usage and terminate if exceeded:

{
  "runtime": {
    "cpu_limit_percent": 50.0
  }
}

Behavior:

80% threshold: Warning logged
100% threshold: Process terminated

Example Logs:

WARN Node 'camera' CPU usage at 82.5% of limit (41.2% / 50.0%)
ERROR Node 'camera' exceeded CPU limit (52.3% / 50.0%) - terminating

Resource Monitoring

Resource checks run during health check intervals:

# Check every 30 seconds
mecha10-node-runner --health-check-interval 30 camera

Implementation:

Uses sysinfo crate for cross-platform monitoring
Works on Linux, macOS, and Windows
Minimal performance overhead

Dependency Ordering

Ensure nodes start in correct order by waiting for dependencies:

{
  "depends_on": ["redis", "database"],
  "startup_timeout_secs": 30
}

How it works:

Node-runner checks for each dependency
Polls Redis key /_internal/node/{dep}/alive every 500ms
Waits up to startup_timeout_secs for each dependency
Fails with clear error if timeout exceeded

Example Output:

Waiting for dependencies: redis, database
✓ redis is ready
✓ database is ready
Starting node...

Error Example:

❌ Dependency timeout: 'database' not ready after 30 seconds

Possible solutions:
1. Check that 'database' node is running
2. Increase startup_timeout_secs in config
3. Verify Redis connectivity

Troubleshooting

Pre-flight Check Failures

Redis not reachable:

# Check Redis status
docker compose ps redis

# Start Redis
docker compose up -d redis

# Test connection
redis-cli ping

Insufficient disk space:

# Check available space
df -h

# Clean up old logs
rm logs/*.log.old

# Or reduce requirement in config

Port already in use:

# Find process using port
lsof -i :8080  # Linux/Mac
netstat -ano | findstr :8080  # Windows

# Kill process or use different port

Permission denied:

# Create logs directory
mkdir -p logs

# Fix permissions
chmod 755 logs

# Or run with appropriate user

GPU not found:

# Check GPU drivers (NVIDIA)
nvidia-smi

# Check CUDA installation
nvcc --version

# Or set require_gpu: false for CPU-only

Restart Loop Debugging

If a node is restarting repeatedly:

1. Check node logs:

tail -f logs/camera.log

2. Disable restart temporarily:

{
  "runtime": {
    "restart_policy": "never"
  }
}

3. Run node directly:

# Run node without supervisor to see errors
target/release/camera

4. Common causes:

Configuration error (fix config.json)
Missing hardware (check device is connected)
Permission denied (run with appropriate privileges)
Resource exhaustion (increase limits or reduce load)

Resource Limit Issues

Node killed for exceeding memory:

Check current usage:

# Monitor node memory
ps aux | grep camera

Solutions:

Increase limit if usage is legitimate
Investigate memory leak if unexpected
Optimize node code to use fewer resources

Node killed for exceeding CPU:

Check current usage:

# Monitor node CPU
top -p $(pgrep camera)

Solutions:

Increase limit if usage is legitimate
Profile node to find CPU hotspots
Optimize expensive operations

Dependency Timeout

Database dependency not ready after 30s:

Check dependency status:

# Is the node running?
ps aux | grep database

# Check logs
tail -f logs/database.log

Solutions:

Increase startup_timeout_secs in config
Fix dependency node startup issues
Verify Redis connectivity for readiness checks

Model Download Failures

Model download fails from HuggingFace:

Check internet connection:

curl -I https://huggingface.co

Verify model paths:

{
  "models": {
    "vision_model": {
      "repo": "Ultralytics/YOLOv8",  # Check this exists
      "file": "yolov8n.onnx"         # Check filename is correct
    }
  }
}

Solutions:

Check internet connectivity
Verify HuggingFace repo/file paths
Download manually and place in models/ directory

Future enhancements

Health checks before start (Redis, GPU, network)
Resource limits (CPU, memory)
Telemetry hooks
Dependency checking (ensure other nodes are running)
GPU allocation

Remote Node Control (Dashboard/API)

When running in development mode (mecha10 dev), nodes can be controlled remotely via the dashboard or API:

Stop via Dashboard

When you click "Stop" on a node in the dashboard:

Dashboard publishes a stop command to Redis (system/cli/command)
CLI's CommandListener receives the command
CLI sends SIGTERM to the node-runner process group
Node-runner's supervisor receives SIGTERM and forwards it to the child node
Both node-runner and child process exit cleanly
No restart occurs - SIGTERM signals intentional shutdown

Important: Even if a node has restart_policy: "always", stopping via dashboard will NOT trigger a restart. The supervisor only restarts nodes that exit unexpectedly, not those terminated via signal.

Start via Dashboard

When you click "Start" on a stopped node:

Dashboard publishes a start command to Redis
CLI's CommandListener receives the command
CLI spawns a new node-runner process (or direct node process in standalone mode)
Node starts with configured restart policy

Restart via Dashboard

When you click "Restart":

Stop command is sent (as above)
Brief delay for cleanup (500ms)
Start command is sent (as above)

When Restart Policies DO Apply

Restart policies trigger when:

Node process crashes (non-zero exit code)
Node process terminates unexpectedly
Parent node-runner is still running

Restart policies do NOT trigger when:

CLI sends SIGTERM (stop command from dashboard)
User presses Ctrl+C in terminal
mecha10 dev session ends
Node-runner itself is terminated

Troubleshooting Unexpected Restarts

If a node seems to restart after being stopped:

1. Check if the correct process was stopped:

# View running processes
ps aux | grep mecha10

# The node-runner should NOT be running after stop

2. Check restart policy:

// configs/dev/nodes/<node>/config.json
{
  "runtime": {
    "restart_policy": "never"  // Should be "never" to prevent restarts
  }
}

3. Check CLI logs:

# The system log panel in dev mode shows:
# 🛑 [Dashboard] Stopping node 'speaker'...
# ✅ [Dashboard] Stopped node 'speaker'

4. Check command-listener debug log:

cat /tmp/command-listener.log

mecha10-runtime - In-process node supervision library
mecha10-cli - Development CLI
mecha10-launcher - Fleet management service

mecha10-node-runner 0.8.0

mecha10-node-runner

Overview

Why?

Usage

Direct execution

Via CLI (development)

Via Launcher (edge)

How it works

Architecture

Dependencies

Restart Policies

1. Never (default)

2. On-Failure

3. Always

Health Monitoring

Configuration File

Pre-flight Checks

Redis Connectivity

Disk Space

Port Availability

File Permissions

GPU Detection

Resource Limits

Memory Limits

CPU Limits

Resource Monitoring

Dependency Ordering

Troubleshooting

Pre-flight Check Failures

Restart Loop Debugging

Resource Limit Issues

Dependency Timeout

Model Download Failures

Future enhancements

Remote Node Control (Dashboard/API)

Stop via Dashboard

Start via Dashboard

Restart via Dashboard

When Restart Policies DO Apply

Troubleshooting Unexpected Restarts

Related