# Zinit Rhai Integration - Technical Documentation
This document explains the complete architecture of Zinit's Rhai scripting integration, including the socket protocol, real-time output streaming, and how all the pieces fit together.
## Architecture Overview
```
┌─────────────────┐ Unix Socket ┌─────────────────────┐
│ zinit client │◄───────────────────────────►│ zinit daemon │
│ (CLI tool) │ ~/hero/var/zinit.sock │ (background) │
└─────────────────┘ └─────────────────────┘
│ │
│ sends script │ executes script
│ receives streamed output │ via Rhai engine
▼ ▼
┌─────────────────┐ ┌─────────────────────┐
│ script.rhai │ │ ZinitEngine │
│ or stdin │ │ (Rhai + zinit) │
└─────────────────┘ └─────────────────────┘
```
## Key Components
### 1. Fixed Paths
Zinit uses hardcoded paths - no configuration needed:
- **Config directory**: `~/hero/cfg/zinit/` - Service YAML files
- **Socket file**: `~/hero/var/zinit.sock` - Unix domain socket for RPC
- **Binary**: `~/hero/bin/zinit` - The zinit executable
Only ONE zinit instance can run at a time. The socket file acts as a lock.
### 2. Daemon Modes
```bash
zinit start # Foreground - blocks, prints to terminal
zinit start -bg # Background - daemonizes via double-fork
```
Background mode uses Unix double-fork to properly daemonize:
1. First fork - parent waits for socket to be ready
2. Second fork - creates session leader, detaches from terminal
3. Child runs the actual daemon with tokio runtime
**Important**: The fork must happen BEFORE creating the tokio runtime, as tokio doesn't survive fork.
### 3. Socket Protocol
The protocol is simple line-based with a delimiter:
**Client → Server:**
```
<script line 1>
<script line 2>
...
===
```
The `===` (two or more `=` characters) signals "execute now".
**Server → Client:**
```
ok
<streamed output line 1>
<streamed output line 2>
...
=====
```
Or on error:
```
ok
<any output before error>
ERROR: <error message>
=====
```
The `=====` marks end of response.
## Real-Time Output Streaming
### The Problem
Rhai scripts run synchronously in a blocking context. Without streaming, all `print()` output would be buffered and only returned after the script completes.
### The Solution
We use a callback mechanism with channels to stream output as it happens:
```
┌──────────────┐ callback ┌──────────────┐ channel ┌──────────────┐
│ Rhai Engine │───────────────►│ Output Sender │─────────────►│ Socket Writer│
│ print() │ │ (closure) │ │ (async task)│
└──────────────┘ └──────────────┘ └──────────────┘
```
### Implementation Details
#### 1. Global Output Sender (`src/rhai_api/engine.rs`)
```rust
// Type alias for the output sender callback
type OutputSender = Box<dyn Fn(&str) + Send + Sync>;
lazy_static::lazy_static! {
static ref OUTPUT_SENDER: Mutex<Option<OutputSender>> = Mutex::new(None);
static ref OUTPUT_BUFFER: Mutex<String> = Mutex::new(String::new());
}
/// Set callback for real-time output
pub fn set_output_sender<F>(sender: F)
where
F: Fn(&str) + Send + Sync + 'static,
{
if let Ok(mut guard) = OUTPUT_SENDER.lock() {
*guard = Some(Box::new(sender));
}
}
/// Clear the output sender
pub fn clear_output_sender() {
if let Ok(mut guard) = OUTPUT_SENDER.lock() {
*guard = None;
}
}
/// Called by Rhai's on_print callback
fn append_output(s: &str) {
if let Ok(guard) = OUTPUT_SENDER.lock() {
if let Some(ref sender) = *guard {
sender(s);
return;
}
}
if let Ok(mut buf) = OUTPUT_BUFFER.lock() {
buf.push_str(s);
buf.push('\n');
}
}
```
#### 2. Rhai Engine Print Hooks (`src/rhai_api/engine.rs`)
```rust
fn register_utility_functions(engine: &mut Engine) {
engine.on_print(|s| {
append_output(s);
});
engine.on_debug(|s, source, pos| {
let location = if let Some(src) = source {
format!("[{}:{}] ", src, pos)
} else if !pos.is_none() {
format!("[{}] ", pos)
} else {
String::new()
};
append_output(&format!("[DEBUG] {}{}", location, s));
});
}
```
#### 3. Socket Handler with Streaming (`src/rhai_api/socket_server.rs`)
```rust
async fn handle_connection(stream: UnixStream, zinit: Arc<RwLock<ZInit>>) {
writer.write_all(b"ok\n").await?;
let (tx, rx) = std::sync::mpsc::channel::<String>();
// Spawn async task to write output as it arrives
let writer_clone = writer.clone();
let output_writer = tokio::spawn(async move {
while let Ok(output) = rx.recv() {
let mut w = writer_clone.lock().await;
w.write_all(output.as_bytes()).await;
w.flush().await; }
});
// Execute script with streaming callback
execute_script_streaming(&script, zinit, tx).await;
// Wait for output writer to finish, send delimiter
output_writer.await;
writer.write_all(b"=====\n").await;
}
async fn execute_script_streaming(
script: &str,
zinit: Arc<RwLock<ZInit>>,
output_tx: std::sync::mpsc::Sender<String>,
) {
tokio::task::spawn_blocking(move || {
let tx = output_tx.clone();
super::engine::set_output_sender(move |s: &str| {
tx.send(format!("{}\n", s));
});
engine.run(&script);
super::engine::clear_output_sender();
}).await;
}
```
#### 4. Client Real-Time Display (`src/rhai_api/socket_server.rs`)
```rust
pub async fn send_script(socket_path: &Path, script: &str) -> Result<String> {
stream.write_all(script.as_bytes()).await?;
stream.write_all(b"\n===\n").await?;
reader.read_line(&mut status_line).await?;
loop {
reader.read_line(&mut line).await?;
if is_execute_marker(&line) { break; }
print!("{}", line);
std::io::stdout().flush();
}
}
```
## Auto-Start Feature
When running `zinit script.rhai`, zinit automatically starts if not running:
```rust
fn ensure_zinit_running() -> bool {
if ping(&socket_path) {
return true;
}
Command::new("zinit")
.args(["start", "-bg"])
.spawn();
for _ in 0..30 {
sleep(100ms);
if ping(&socket_path) {
return true;
}
}
false
}
```
## Logging Architecture
Zinit has its own logging for services (separate from Rhai `print()`):
### Service Log Modes
```rhai
zinit_service_new()
.log("ring") // Ring buffer (default) - keeps last N lines in memory
.log("stdout") // Forward to zinit's stdout
.log("none") // Discard all output
```
### Rhai Output vs Service Logs
| Source | Destination | When |
|--------|-------------|------|
| `print()` in Rhai | Streamed to client via socket | During script execution |
| Service stdout/stderr | Ring buffer or zinit stdout | While service runs |
| `zinit_status_md()` | Return value to client | When called |
## File Structure
```
src/
├── main.rs # CLI entry point, argument parsing
├── app.rs # Paths, initialization, daemonization
├── lib.rs # Library exports
├── instructions.md # AI agent instructions (embedded)
└── rhai_api/
├── mod.rs # Module exports
├── engine.rs # ZinitEngine, print capture, function registration
├── socket_server.rs # Unix socket server, streaming protocol
├── service_builder.rs # ServiceBuilder pattern for Rhai
└── types.rs # Type definitions
```
## Dependencies
All key crates used in zinit:
```toml
[dependencies]
# Core
anyhow = "1.0" # Error handling
tokio = { version = "1.44", features = ["full", "net", "io-util", "sync"] } # Async runtime
tokio-stream = "0.1.17" # Async streams
# Rhai Scripting
rhai = { version = "1.19", features = ["sync", "serde"] } # Scripting engine
lazy_static = "1.4" # Global state for output sender
# Interactive REPL (TUI)
rustyline = { version = "14.0", features = ["derive"] } # Readline with completion/history
colored = "2.1" # Terminal colors for syntax highlighting
# Unix/System
nix = "0.22.1" # Unix-specific (fork, signals, processes)
libc = "0.2" # Low-level C bindings
command-group = "1.0.8" # Process group management
sysinfo = "0.29.10" # System/process information (CPU, memory)
# Serialization
serde = { version = "1.0", features = ["derive"] } # Serialization framework
serde_yaml = "0.8" # YAML config files
serde_json = "1.0" # JSON support
# Utilities
dirs = "5.0" # Home directory detection
shlex = "1.1" # Shell-like argument parsing
clap = { version = "4.0", features = ["derive"] } # CLI argument parsing
fern = "0.6" # Logging
log = "0.4" # Logging facade
thiserror = "1.0" # Error derive macros
git-version = "0.3.5" # Embed git version
```
### Key Library Purposes
| Library | Purpose |
|---------|---------|
| `rhai` | Embedded scripting language for service configuration |
| `rustyline` | Interactive REPL with readline-style editing, tab completion, history |
| `colored` | Terminal color output for syntax highlighting |
| `tokio` | Async runtime for socket server and concurrent operations |
| `lazy_static` | Global mutable state for output streaming callback |
| `nix` | Unix process control (fork, signals, process groups) |
| `sysinfo` | Query CPU/memory usage of services |
## Common Patterns
### Blocking Rhai in Async Context
Rhai is synchronous, but zinit uses tokio. We bridge them with `spawn_blocking`:
```rust
tokio::task::spawn_blocking(move || {
let rt = tokio::runtime::Handle::current();
let result = rt.block_on(async {
zinit.read().await.some_method().await
});
engine.run(&script)
}).await
```
### Macro for Async Zinit Functions
```rust
macro_rules! block_on {
($fut:expr) => {
tokio::task::block_in_place(||
tokio::runtime::Handle::current().block_on($fut)
)
};
}
// Usage in registered function
engine.register_fn("zinit_start", move |name: &str| -> bool {
let z = zinit.clone();
block_on!(async {
let zinit = z.read().await;
zinit.start(&name).await.is_ok()
})
});
```
## Testing
### Manual Testing
```bash
# Terminal 1: Start daemon in foreground to see logs
zinit start
# Terminal 2: Run scripts
zinit rhaiexamples/01_full_test.rhai
# Or test streaming
echo 'print("line 1"); sleep(1); print("line 2");' | zinit -i
```
### Verify Streaming
Create a test script with delays:
```rhai
// rhaiexamples/02_stream_test.rhai
print("Starting...");
sleep(1);
print("1 second");
sleep(1);
print("2 seconds");
print("Done!");
```
Run it and observe output appears line-by-line with delays, not all at once.
## Troubleshooting
### Socket Issues
```bash
# Check if zinit is running
ls -la ~/hero/var/zinit.sock
# Kill stale instance
pkill -9 zinit
rm -f ~/hero/var/zinit.sock
# Start fresh
zinit start -bg
```
### Output Not Streaming
1. Check `flush()` is called after each write
2. Verify output sender callback is set before script runs
3. Check channel isn't dropping messages
### Function Not Found
Ensure the function is registered with correct argument types. Rhai is strongly typed - `zinit_stop("name")` won't match `zinit_stop()` (no args).
## Interactive REPL (TUI Shell)
The `zinit --ui` command starts an interactive shell with advanced features.
### Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ zinit --ui │
├─────────────────────────────────────────────────────────────┤
│ rustyline (readline) │
│ ├── Tab Completion (ReplHelper) │
│ ├── Syntax Highlighting (Highlighter trait) │
│ ├── History (~/.zinit_history) │
│ └── Hints (function signatures) │
├─────────────────────────────────────────────────────────────┤
│ REPL Commands (/help, /load, /functions, etc.) │
├─────────────────────────────────────────────────────────────┤
│ Socket Client (send_script) │
│ └── Streams output to terminal in real-time │
└─────────────────────────────────────────────────────────────┘
```
### Key Components (`src/repl.rs`)
#### 1. ReplHelper Struct
Implements rustyline traits for completion, highlighting, and hints:
```rust
#[derive(Helper)]
struct ReplHelper {
hinter: HistoryHinter,
}
impl Completer for ReplHelper { ... }
impl Hinter for ReplHelper { ... }
impl Highlighter for ReplHelper { ... }
impl Validator for ReplHelper {}
```
#### 2. Function Definitions
All zinit functions are defined with signatures and descriptions for completion/help:
```rust
const ZINIT_FUNCTIONS: &[(&str, &str, &str)] = &[
// (signature, return_type, description)
("zinit_ping()", "bool", "Check if zinit is responding"),
("zinit_start()", "bool", "Start zinit daemon if not running"),
("zinit_service_new()", "ServiceBuilder", "Create a new service builder"),
// ... more functions
];
const BUILDER_METHODS: &[(&str, &str)] = &[
(".exec(cmd)", "Command to run (required)"),
(".test(cmd)", "Health check command"),
// ... more methods
];
const REPL_COMMANDS: &[(&str, &str)] = &[
("/help", "Show this help"),
("/functions", "List all zinit functions"),
("/load <file>", "Load and execute a .rhai script"),
// ... more commands
];
```
#### 3. Tab Completion
```rust
impl Completer for ReplHelper {
fn complete(&self, line: &str, pos: usize, _ctx: &Context<'_>)
-> rustyline::Result<(usize, Vec<Pair>)>
{
let word = &line[word_start..pos];
if word.starts_with('/') {
for (cmd, desc) in REPL_COMMANDS { ... }
}
if word.starts_with('.') {
for (method, desc) in BUILDER_METHODS { ... }
}
for (func, ret, desc) in ZINIT_FUNCTIONS {
if func_name.starts_with(word) { ... }
}
for kw in RHAI_KEYWORDS { ... }
}
}
```
#### 4. Syntax Highlighting
```rust
impl Highlighter for ReplHelper {
fn highlight<'l>(&self, line: &'l str, _pos: usize) -> Cow<'l, str> {
}
}
fn highlight_word(word: &str) -> String {
if RHAI_KEYWORDS.contains(&word) {
return format!("{}", word.magenta().bold());
}
if word.starts_with("zinit_") {
return format!("{}", word.blue().bold());
}
}
```
#### 5. Multi-line Input Detection
```rust
let open_braces = buffer.matches('{').count();
let close_braces = buffer.matches('}').count();
if open_braces > close_braces {
continue;
}
if line.ends_with('\\') {
continue;
}
```
#### 6. Script Loading
```rust
async fn load_script(file_path: &str, socket_path: &Path) -> Result<()> {
let expanded_path = if file_path.starts_with("~/") {
dirs::home_dir().map(|h| h.join(&file_path[2..]))
} else {
PathBuf::from(file_path)
};
let script = std::fs::read_to_string(&expanded_path)?;
send_script(socket_path, &script).await
}
```
### Dependencies for REPL
```toml
[dependencies]
rustyline = { version = "14.0", features = ["derive"] }
colored = "2.1"
```
### Adding a New REPL Command
1. Add to `REPL_COMMANDS` constant:
```rust
const REPL_COMMANDS: &[(&str, &str)] = &[
("/mycommand", "Description of my command"),
];
```
2. Handle in the match statement in `run_repl()`:
```rust
match line {
"/mycommand" => {
println!("My command executed!");
continue;
}
}
```
## Service Name Normalization
All service names are normalized to prevent confusion:
```rust
fn normalize_service_name(name: &str) -> String {
name.replace('-', "_").to_lowercase()
}
```
This means:
- `Test-Service` → `test_service`
- `MY_SERVICE` → `my_service`
- `web-server` → `web_server`
Applied to all functions: `zinit_monitor`, `zinit_start`, `zinit_stop`, `zinit_status`, etc.
## Extending
### Adding a New Rhai Function
1. Add to `register_zinit_functions()` in `engine.rs`:
```rust
let z = zinit.clone();
engine.register_fn("zinit_my_func", move |arg: &str| -> bool {
let z = z.clone();
let name = normalize_service_name(arg); block_on!(async {
let zinit = z.read().await;
zinit.my_method(&name).await.is_ok()
})
});
```
2. Add to `ZINIT_FUNCTIONS` in `repl.rs` for completion:
```rust
("zinit_my_func(name)", "bool", "Description of my function"),
```
3. Document in `src/instructions.md`
### Adding a Builder Method
1. Add field to `ServiceBuilder` in `service_builder.rs`
2. Add method in `rhai_fns` module
3. Register in `register_types()`:
```rust
.register_fn("my_option", rhai_fns::service_my_option)
```
4. Add to `BUILDER_METHODS` in `repl.rs`:
```rust
(".my_option(value)", "Description of option"),
```