<p align="center">
<img src="https://raw.githubusercontent.com/cyber-boost/helix/refs/heads/master/media/logo.png" alt="Helix Logo" width="400"/>
</p>
# Helix Configuration - Configuration Without the Headaches
## Overview
Helix Configuration is a purpose-built configuration language designed for AI systems. No more TOML limitations, JSON verbosity, or environment variable chaos. This is configuration that understands AI workflows, agents, and pipelines natively. Designed and built for Maestro.ps
**Current Status**: W.I.P. not production-ready with but do have a full compiler, CLI tools, and comprehensive language features.
## Why Helix Configuration?
Crates install: cargo build --release --features full && hlx install
### The Problems We Solved
**TOML Problems:**
- No native support for complex structures
- Arrays of tables are a nightmare
- No conditionals or references
- Limited type system
- Can't express workflows or pipelines
**JSON Problems:**
- Verbose and unreadable
- No comments (seriously?)
- Everything is a string or number
- No duration types (30m → "30m" → parse → pray)
- Trailing comma hell
**ENV Variables Problems:**
- Everything is a string
- No structure or hierarchy
- No validation
- Scattered across shell scripts
- No version control friendly
**YAML Problems:**
- Whitespace sensitivity disasters
- Norway problem (no: false)
- Ambiguous types
- Anchors and references are cryptic
- Multi-line strings are painful
### The helix Solution
**Built for AI Configuration:**
- **Native AI constructs** - agents, workflows, pipelines, crews
- **Rich type system** - durations (30m), references ($VAR), tags (@memory.key)
- **Hierarchical structure** - Clean nesting without the pain
- **Comments** - Because documentation matters
- **Validation** - Catch errors at compile time, not runtime
- **Binary compilation** - Parse once, load instantly
## Language Features
### Basic Syntax
```helix
# Comments start with #
project "helixia" {
version = "3.0.0"
author = "B"
description = "AI-Human collaboration system"
}
```
### Agent Definition
```helix
agent "senior-rust-engineer" {
model = "claude-3-opus"
role = "Systems Architect"
temperature = 0.7
max_tokens = 100000
capabilities [
"rust-async"
"memory-optimization"
"concurrency"
"zero-copy"
]
backstory {
15 years of systems programming
Rust contributor since 2015
Focus on safety and performance
Built high-frequency trading systems
}
tools = [
"cargo"
"rustc"
"clippy"
"miri"
]
}
```
### Workflow Definition
```helix
workflow "code-review-pipeline" {
trigger = "pull_request"
step "analyze" {
agent = "senior-rust-engineer"
task = "Review code for safety and performance"
timeout = 30m # Native duration type!
parallel = false
depends_on = []
}
step "test" {
crew = ["test-engineer", "qa-engineer"]
task = "Run comprehensive test suite"
timeout = 1h
parallel = true
depends_on = ["analyze"]
retry {
max_attempts = 3
delay = 30s
backoff = "exponential"
}
}
pipeline {
analyze -> test -> approve -> merge
}
}
```
### Memory Configuration
```helix
memory {
provider = "helix_db" # Our AI-native database
connection = "file:./data/agents.db"
embeddings {
model = "text-embedding-3-small"
dimensions = 1536
batch_size = 100
}
cache {
size = 1000
ttl = 24h # Duration type
}
}
```
### Context Management
```helix
context "production" {
environment = "prod"
debug = false
max_tokens = 100000
variables {
api_endpoint = "https://api.helix.cm"
timeout = 30s
retry_count = 3
}
secrets {
anthropic_key = $ANTHROPIC_API_KEY # Environment reference
openai_key = $OPENAI_API_KEY
database_url = "vault:database/prod/url" # Vault reference
}
}
```
### Crew Definition
```helix
crew "development-team" {
agents [
"senior-rust-engineer"
"code-reviewer"
"test-engineer"
]
process = "hierarchical"
manager = "senior-rust-engineer"
max_iterations = 10
verbose = true
}
```
## Type System
### Primitive Types
```helix
string_value = "Hello, World"
number_value = 42
float_value = 3.14
boolean_value = true
null_value = null
```
### Duration Types
```helix
# All of these work naturally
timeout = 30s # 30 seconds
delay = 5m # 5 minutes
cache_ttl = 24h # 24 hours
retention = 7d # 7 days
```
### References
```helix
# Environment variables
api_key = $API_KEY
# Memory references
context = @memory.conversation.latest
# Variable references
base_url = ${config.api.endpoint}
```
### Environment Variables
```helix
# Pull from shell environment, .bashrc, or .env files
agent "my-agent" {
model = $ANTHROPIC_API_KEY
tools = ["tool1", "tool2"]
}
context "production" {
secrets {
# Environment variables
db_password = $DATABASE_PASSWORD
api_key = $MY_API_KEY
# Vault references (for sensitive data)
cert_path = "vault:ssl/certificate"
private_key = "vault:ssl/private_key"
}
variables {
# Regular configuration values
api_endpoint = "https://api.production.com"
timeout = 30s
max_retries = 3
}
}
```
**Setting up Environment Variables:**
```bash
# In your .bashrc, .zshrc, or .env file
export ANTHROPIC_API_KEY="your-key-here"
export DATABASE_PASSWORD="your-password"
export MY_API_KEY="another-key"
# helix will automatically pick these up
```
### Arrays
```helix
# Simple arrays
tags = ["rust", "systems", "performance"]
# Multi-line arrays
capabilities [
"reasoning"
"generation"
"analysis"
]
```
### Objects
```helix
# Inline objects
metadata = { version = "1.0", stable = true }
# Nested objects
config {
api {
endpoint = "https://api.example.com"
timeout = 30s
}
}
```
### Special Constructs
**Pipeline Flow:**
```helix
pipeline {
fetch -> process -> validate -> store
}
```
**Hierarchical Tags:**
```helix
tags [
"capability:reasoning:logical"
"model:gpt-4"
"context:conversation"
]
```
**Weighted Values:**
```helix
priority = "high:0.9"
confidence = "certain:0.95"
```
## Parser Architecture
```
helix Source Code
↓
[Lexer] → Tokens
↓
[Parser] → AST
↓
[Validator] → Validated AST
↓
[Compiler] → Binary Format
```
### Lexer
Converts text into tokens:
- Keywords (agent, workflow, context, etc.)
- Identifiers
- Literals (strings, numbers, durations)
- Operators (=, ->, [, ], {, })
### Parser
Builds Abstract Syntax Tree (AST):
- Declarations (agents, workflows, contexts)
- Expressions (values, references, arrays)
- Statements (assignments, blocks)
### Validator
Ensures correctness:
- Type checking
- Reference validation
- Dependency resolution
- Constraint verification
## Usage
### In Rust Code
```rust
use helix_config::{parse, helixConfig};
// Parse from string
let config_str = r#"
agent "assistant" {
model = "gpt-4"
temperature = 0.7
}
"#;
let config = parse(config_str)?;
let agent = config.agents.get("assistant").unwrap();
```
### File Loading
```rust
use helix_config::helixLoader;
let mut loader = helixLoader::new();
// Load single file
let config = loader.load_file("config.hlxbb")?;
// Load directory of .hlxbb files
loader.load_directory("./configs")?;
// Access merged configuration
let merged = loader.get_merged_config();
```
### With Validation
```rust
use helix_config::{parse_and_validate, ValidationRules};
let rules = ValidationRules {
require_version: true,
max_agents: Some(100),
allowed_models: vec!["gpt-4", "claude-3"],
};
let config = parse_and_validate(source, rules)?;
```
## Language Reference
### Keywords
```
agent - Define an AI agent
workflow - Define a workflow
context - Define an execution context
memory - Configure memory/storage
crew - Define an agent crew
pipeline - Define a processing pipeline
step - Workflow step
trigger - Workflow trigger
capabilities - Agent capabilities
backstory - Agent background
secrets - Sensitive configuration
embeddings - Embedding configuration
```
### Operators
```
= - Assignment
-> - Pipeline flow
[] - Array delimiter
{} - Block/object delimiter
$ - Environment variable
@ - Memory reference
# - Comment
: - Type/weight separator
```
### Built-in Functions (Future)
```helix
# Planned for future versions
result = sum([1, 2, 3])
encoded = base64("data")
hashed = sha256("content")
```
## Best Practices
### 1. Organization
```helix
# Group related configurations
# agents.hlxbb
agent "coder" { ... }
agent "reviewer" { ... }
# workflows.hlxbb
workflow "ci" { ... }
workflow "cd" { ... }
# config.hlxbb
memory { ... }
context "prod" { ... }
```
### 2. Naming Conventions
```helix
# Use descriptive names
agent "senior-rust-engineer" # Good
agent "sre" # Too short
agent "a1" # Meaningless
# Use consistent separators
workflow "code-review-pipeline" # kebab-case
context "production_environment" # snake_case (pick one!)
```
### 3. Comments
```helix
# Document why, not what
agent "specialist" {
# Higher temperature for creative problem solving
temperature = 0.9
# Limit tokens to control costs in development
max_tokens = 50000
}
```
### 4. Reusability
```helix
# Define base configurations (future feature)
base_agent {
temperature = 0.7
max_tokens = 100000
}
agent "coder" extends base_agent {
model = "gpt-4"
role = "Developer"
}
```
## Error Messages
Helix Configuration provides clear, actionable error messages:
```
Error at line 15, column 8:
timeout = "30 minutes"
^^^^^^^^^^^^
Expected duration type (e.g., 30m, 1h, 5s)
```
```
Error at line 23:
agent = "undefined-agent"
^^^^^^^^^^^^^^^^^
Reference to undefined agent. Available agents:
- senior-rust-engineer
- code-reviewer
- test-engineer
```
## Tooling
### Syntax Highlighting
Available for:
- VS Code (extension: `helix-config`)
- Vim (plugin: `vim-helix`)
- Sublime Text (package: `helix`)
### Formatter
```bash
# Format .hlxbb files
helix fmt config.hlxbb
# Check formatting
helix fmt --check config.hlxbb
```
### Linter
```bash
# Lint for common issues
helix lint config.hlxbb
# With auto-fix
helix lint --fix config.hlxbb
```
## Migration Guide
### From TOML
```toml
# Before (TOML)
[agent.coder]
model = "gpt-4"
temperature = 0.7
capabilities = ["rust", "python"]
```
```helix
# After (helix)
agent "coder" {
model = "gpt-4"
temperature = 0.7
capabilities ["rust", "python"]
}
```
### From JSON
```json
// Before (JSON)
{
"workflow": {
"name": "ci",
"timeout": "30m",
"steps": [...]
}
}
```
```helix
# After (helix)
workflow "ci" {
timeout = 30m # Native duration!
step { ... }
}
```
### From YAML
```yaml
# Before (YAML)
agent:
name: coder
config:
model: gpt-4
temperature: 0.7
```
```helix
# After (helix)
agent "coder" {
model = "gpt-4"
temperature = 0.7
}
```
## Current Implementation Status
### ✅ Completed Features
- **Full Lexer** with source location tracking, error recovery, and line continuation
- **Recursive Descent Parser** with precedence climbing for expressions
- **AST** with visitor pattern and pretty printing
- **Semantic Analyzer** with type checking, reference validation, and circular dependency detection
- **Code Generator** with IR, optimizations, and binary serialization
- **Binary Compiler** with 4-level optimization pipeline and compression
- **CLI Tool** (`helix`) with 25+ commands including compile, decompile, bundle, validate, test, bench, serve, and more
- **Migration Tools** for JSON, TOML, YAML, and .env files
- **Hot Reload System** with file watching and automatic recompilation
- **Dependency Resolution** with circular dependency detection
- **Comprehensive Testing** including unit, integration, fuzzing, and round-trip tests
- **Performance Benchmarks** validating sub-millisecond parsing for small configs
- **5 Real-World Examples** demonstrating all language features
- **Project Management** with init, add, remove, clean, reset, build, run commands
- **Development Tools** with fmt, lint, generate, publish, sign, export, import
- **System Integration** with config, cache, doctor commands
### 📁 Project Structure
```
helix/
├── Cargo.toml # Package definition with features
├── lib.rs # Public API and exports
├── types.rs # Configuration types
├── lexer.rs # Tokenization with source tracking
├── parser.rs # Recursive descent parser with error recovery
├── ast.rs # Abstract syntax tree and visitor pattern
├── semantic.rs # Semantic analysis and validation
├── codegen.rs # IR generation
├── error.rs # Error handling and types
├── integration.rs # Integration tests
├── tests.rs # Unit test suite
├── benches.rs # Performance benchmarks
├── compiler/ # Binary compilation subsystem
│ ├── mod.rs # Module exports
│ ├── binary.rs # Binary format definitions
│ ├── optimizer.rs # Optimization pipeline (0-3)
│ ├── serializer.rs # Binary serialization
│ ├── loader.rs # Runtime loading with mmap
│ ├── bundle.rs # Multi-file bundling
│ ├── cli.rs # CLI implementation
│ ├── cli/ # CLI command modules
│ ├── config/ # Configuration management
│ ├── project/ # Project management
│ ├── publish/ # Publishing and distribution
│ ├── tools/ # Development tools
│ └── workflow/ # Workflow management
├── src/bin/
│ └── helix.rs # CLI binary entry point
├── examples/ # 5 complete .hlxbb example files
│ ├── ai_development_team.hlxbb
│ ├── data_pipeline.hlxbb
│ ├── research_assistant.hlxbb
│ ├── customer_support.hlxbb
│ └── minimal.hlxbb
├── binaries/ # Compiled binary examples
├── admin/ # Documentation and scripts
├── summaries/ # Development summaries
└── build.sh # Build and test script
## Performance (Actual Benchmarks)
### Parse Performance
| Small (100 lines) | ~0.5ms | ~0.3ms | ~0.8ms | <1ms |
| Medium (1K lines) | ~45ms | ~28ms | ~72ms | <10ms |
| Large (10K lines) | ~450ms | ~280ms | ~750ms | <100ms |
### With Binary Compilation
| Parse | 35ms | N/A | N/A |
| Load | 35ms | 0.8ms | 44x |
| Validate | 12ms | 0ms | ∞ |
| Total | 47ms | 0.8ms | 59x |
## Installation & Usage
### As a Library
```toml
[dependencies]
helix-config = { path = "path/to/helix" }
# Or with specific features:
helix-config = { path = "path/to/helix", default-features = false } # Just parsing
helix-config = { path = "path/to/helix", features = ["compiler"] } # With compiler
```
### CLI Installation
```bash
cd helix
cargo install --path . --features cli
# Now use helix command globally
helix compile config.hlxbb -O3
```
### CLI Commands
```bash
# Core compilation commands
helix compile config.hlxbb -O3 --compress
helix decompile config.hlxb -o recovered.hlxbb
helix validate config.hlxbb --detailed
helix bundle ./configs/ -o bundle.hlxb --tree-shake
helix optimize config.hlxb -O3
# Project management
helix init my-project
helix add dependency-name
helix remove dependency-name
helix clean
helix reset
helix build
helix run
# Development tools
helix fmt config.hlxbb
helix lint config.hlxbb --fix
helix generate template-name
helix test
helix bench
# System integration
helix watch ./configs/ -O2
helix serve --port 8080
helix info config.hlxb --symbols --sections
helix diff old.hlxb new.hlxb
helix config list
helix cache clear
helix doctor
# Publishing and distribution
helix publish --version 1.0.0
helix sign config.hlxb
helix export --format json
helix import --from toml config.toml
```
## Testing
Run all tests with the provided scripts:
```bash
# Full build and test
./build.sh # Complete build and test suite
# Just test examples
cargo test # Run all tests
cargo test integration # Run integration tests
cargo bench --no-run # Compile benchmarks
cargo build --all-features # Build everything
```
**Note**: Some integration tests may have compilation issues that need to be resolved. The core functionality is working as demonstrated by the successful CLI commands.
## Future Features
### Planned Enhancements
- [ ] Import statements for modular configs
- [ ] Template system with inheritance
- [ ] Conditional compilation
- [ ] Macros for code generation
- [ ] Type aliases
- [ ] Custom validators
- [ ] Schema definitions
## Contributing
Priority areas for contribution:
- Language features for AI workflows
- Performance optimizations
- Better error messages
- IDE integrations
- Documentation examples
## License
BBL - Configuration should enable, not constrain.
## Contributing
Priority areas for contribution:
- Language features for AI workflows
- Performance optimizations
- Better error messages
- IDE integrations
- Documentation examples
## Current Issues
- Some integration tests have compilation issues that need to be resolved
- The project is in active development with ongoing improvements
- CLI commands are fully functional despite test issues