somatize-runtime 0.4.0

Soma (σῶμα — body) is a computational graph runtime for research pipelines, agent orchestration, and data virtualization. Written in Rust with Python bindings.

Part of the Nous-Soma-Chronos ecosystem:

• Nous: Understands, reasons — research IDE, agent graphs, automation
• Soma (this project): Executes, materializes — graphs, optimization, distributed workers
• ChronosVector: Remembers — temporal vector database

Docs: manucouto1.github.io/soma — quickstart, tutorial notebooks, Python API.

Examples: examples/ is a submodule — longer, complete programs rather than feature demonstrations, with every figure produced by the code beside it. Clone it with the repository:

git clone --recursive https://github.com/manucouto1/soma
# already cloned?
git submodule update --init

Key Concepts

Workspace (11 crates)

soma-macros      → proc macro (#[derive(SomaFilter)])
soma-core        → types + traits: Filter, Value, Graph, Event, Schema, Study,
                   DataStore (Local/S3/Zarr), VirtualValue, tracking schema, GraphOverlay
soma-compiler    → Graph → ExecutionPlan (caching, parallelism, distribution)
                   Scheduler, plan visualization
soma-runtime     → GraphSession, executor, NodeCatalog (filters AND steps), caches,
                   samplers, pruners, EffectDriver + journal,
                   StudyRunner, PbtRunner, LocalTracker + RunReader (run directories)
soma-memory      → KnowledgeBase trait + MemoryKB + ChronosKB
soma-worker      → Worker, Protocol, EnvManager, token auth, CLI binary
soma-coordinator → worker registry, routing, heartbeat monitoring
soma-agent       → Research agent loop (observe → hypothesize → experiment → conclude)
soma-mcp         → MCP server (13 tools for code, execution, knowledge)
soma-python      → PyO3 bindings: Graph, Study, Run, RunView, soma.viz, Chain/Fork operators
somatize (soma/) → facade crate re-exporting the workspace

Quick Start

# Run all tests (900+: 582 Rust + 325 Python)
cargo test --workspace
cd soma-python && maturin develop && pytest tests/ -v

# With S3/Zarr DataStore
cargo test -p somatize-core --features s3
cargo test -p somatize-core --features zarr

# With ChronosVector
cargo test -p somatize-memory --features chronos

# MCP server
cargo run -p somatize-mcp -- /path/to/project

Python Usage

from soma import Filter, Graph, Study, search

class Scaler(Filter):
    _differentiable = True

    def fit(self, x, y=None):
        return {"mean": sum(x) / len(x)}

    def forward(self, x, state):
        return [v - state["mean"] for v in x]

class Model(Filter):
    lr: float = search(0.001, 1.0, scale="log")

    def fit(self, x, y=None):
        return {"weights": [0.5] * len(x)}

    def forward(self, x, state):
        return [v * w for v, w in zip(x, state["weights"])]

# Build with >> (chain) and | (fork)
g = Graph.somatize(Scaler() >> Model())
g.fit(train_data)
result = g.forward(test_data)

# Complex topologies
g = Graph.somatize(
    (LoadA() >> NormA() | LoadB() >> NormB())
    >> Aggregate()
    >> Backbone()
    >> (HeadA() | HeadB())
)

# Distributed execution
g.add_worker("http://gpu-box:8080", token="sk-xxx", tags=["gpu"])
g.set_coordinator("http://coord:9090", token="sk-xxx")   # or auto-discovery

See your experiments

Everything a run produces lands in a run directory; everything below just reads it back.

# Track a run — architecture snapshot + lossless event log + metrics
with g.track_run("mos-baseline", tags=["mos"]) as run:
    g.fit(train_data)
    run.log("val_f1", evaluate(g), step=0)

# List and inspect runs (crashed runs are detected via stale heartbeat)
for run in soma.runs():
    print(run.id, run.state, run.name)

view = soma.runs()[0]
view.node_timings()         # per-node wall times, durations, cache hits
print(view.to_mermaid())    # architecture annotated with timing/cache/health

In a notebook, evaluating g draws the architecture as an inline SVG diagram (a materialized differentiable filter draws its inner layers, parameter counts included) — no JavaScript, renders anywhere.

graph LR
    scaler["scaler<br/>26ms"]
    model["model<br/>27ms · ⚠ DEAD_CHANNELS(2)"]
    scaler --> model
    class scaler soma_completed
    class model soma_flagged

With the viz extra (pip install 'somatize[viz]') you get interactive Plotly figures and pandas projections:

study.plot_optimization_history()   # objective per trial + best-so-far
study.plot_parallel_coordinate()    # params → objective
study.plot_param_importances()      # Spearman rank correlation
study.plot_timeline()               # trial gantt
study.trials_dataframe()            # pandas table

view.plot_metrics()                 # logged metric curves
view.plot_gantt()                   # where the run's wall time went
view.plot_health()                  # gradient-audit health flags
view.plot_channels("encoder")      # channel-correlation heatmap

And from the shell:

soma runs                            # list tracked runs
soma graph <run_id>                  # annotated mermaid/dot diagram
soma report <run_id> -o report.html  # self-contained HTML report
soma report <run_id> --inline        # fully offline (embeds plotly.js)

soma report packages the annotated DAG, efficiency tiles, metric curves, the full HPO section with trial table, and the health section into one shareable file. A future live GUI reads the same run-directory files and the same embedded JSON shapes — see the Visualization page.

Workers

# Start a worker with auto-detected capabilities
soma-worker --port 8080 --tags gpu,training --token sk-xxx

# With resource limits (Slurm-style)
soma-worker --cpus 4 --memory 8G --gpus 1 --max-concurrent 2

# With coordinator auto-registration
soma-worker --coordinator http://coord:9090 --token sk-xxx --tags gpu

Workers auto-detect CPU cores, RAM, GPUs (nvidia-smi), and Python environments. Each worker creates isolated venv/conda environments per job with incremental dependency updates.

Feature Flags & Extras

• somatize-core/s3 — S3-compatible DataStore (AWS, Backblaze B2, MinIO)
• somatize-core/zarr — Zarr v3 chunked tensor storage with compression
• somatize-memory/chronos — ChronosVector-backed KnowledgeBase
• somatize[viz] (pip) — Plotly figures, DataFrames, HTML reports, rich tables, tqdm progress

License

Elastic License 2.0