# unifier
Filesystem postbox for inter-process communication. Programs share state by writing values to files in a Unix tree, treating the filesystem as a global data structure — the same strategy used by [filesystem-git-issues](../filesystem-git-issues/) (directories are namespaces, files are keys, UUID-named files are messages).
No central server is required.
## Sister project: Jan
Unifier is the **state plane** of a two-daemon multi-agent setup; [jan-cli](../jan-cli/)
is the **execution plane** (cron schedules, process spawn). The two share no code —
`unifier` has no dependency on Jan and no knowledge of Jan concepts — and are coupled
only by three wire artifacts under `.daemon/`: notices Unifier broadcasts on
`events.sock`, tick lifecycle calls Jan makes on `tick.sock`, and ordinary CLI calls
agents make on `unifier.sock`. Unifier runs perfectly well with no scheduler attached.
The normative schema, delivery semantics (notices are best-effort hints; the board is
the durable truth), and invariants live in
[jan-cli/docs/maintainers/ipc-contract.md](../jan-cli/docs/maintainers/ipc-contract.md).
Changing `Notice` in `src/daemon/notify.rs` or the tick verbs in
`src/daemon/protocol.rs` is a cross-project change.
## State layout
Default root: `~/.local/unifier` (override with `$UNIFIER_HOME` or `--home`).
```
~/.local/unifier/
keys/<path> # persistent key-value (global)
mailbox/<recipient>/<uuid>.txt # point-to-point messages
cron/<min>_<hour>_<dom>_<mon>_<dow>/<uuid>.txt # scheduled drops
sqlite/<name>.sqlite # managed SQLite databases
sqlite/triples.sqlite # reserved subject/predicate/object store
events/<uuid>.json # JSON events (payload + created_at + expires_at)
logs/<uuid>.json # span-tree log entries
.daemon/
unifier.sock # control plane (put/get/mail/…)
tick.sock # tick lifecycle only (Jan → Unifier)
events.sock # outbound notices (Unifier → Jan / watchers)
www/<name> # temp files served over HTTP
http.port # bound localhost web port
chroots/
<name>/
keys/
mailbox/
cron/
sqlite/ # isolated subtree (--chroot)
```
Cron schedule directories use five underscore-separated fields. Use `*` for “any” (e.g. `0_9_*_*_*` = 09:00 daily).
## CLI reference
| `put <key> <value>` | Write a persistent key under `keys/` |
| `get <key>` | Read a key |
| `del <key>` | Delete a key |
| `send [--from <agent>] <recipient> <message>` | Drop a mailbox envelope (id, from, to, payload) |
| `message --from <agent> <recipient> '<json>'` | Structured agent mailbox message + event-socket wakeup |
| `event '<json>' [--ttl <secs>]` | Post a JSON event under `events/` (default TTL 24h; `0` = never) |
| `daemon watch` | Print wakeup notices from `.daemon/events.sock` (for Jan cron) |
| `cron <schedule> <message>` | Schedule a cron message |
| `poll <recipient> [--ack]` | Collect mailbox messages |
| `poll-cron [--ack]` | Collect messages whose schedule matches now |
| `list <path>` | List message files under a subtree |
| `ack <uuid\|path>` | Remove a processed message |
| `root` | Print the effective state root |
| `tick start [label]` | Begin ACID tick (via `.daemon/tick.sock` when daemon is up) |
| `tick phase <name>` | Set active tick phase; broadcasts `{kind:tick,…}` on `events.sock` |
| `tick end` | Commit current tick |
| `tick status` | Show committed/active tick, phase, queue, locks |
| `tick lock` / `tick unlock` | Lock keys for the active tick |
| `daemon start` | Start hot in-memory daemon (background) |
| `daemon stop` | Stop daemon (flushes dirty state) |
| `daemon gc [--dry-run]` | Kill orphan daemons whose `--home` is missing |
| `daemon status` | Show whether daemon is running |
| `daemon flush` | Write dirty in-memory state to disk |
| `chroot init <name>` | Create `chroots/<name>/` with `keys/`, `mailbox/`, `cron/` |
| `chroot list` | List chroot names |
| `namespace set <name>` | Bind a key prefix to this process (and descendants) |
| `namespace get` | Print the effective namespace |
| `namespace clear` | Remove this process's namespace binding |
| `sql list` | List managed SQLite databases under `sqlite/` |
| `sql create <name>` | Create `sqlite/<name>.sqlite` |
| `sql tables <db> [--schema]` | List tables (and columns) in a database |
| `sql exec <db> '<sql>'` | Run SQL (creates the DB file if missing) |
| `triple add <s> <p> <o>` | Insert a triple into the reserved `triples` DB |
| `triple query [-s] [-p] [-o]` | Query triples with optional filters |
| `log start <name> [--parent <id>] [-f k=v]` | Start a new span; prints its UUID |
| `log end <id>` | End a span (record `ended_at`) |
| `log event <span> <message> [-f k=v]` | Append a log event to a span |
| `log field <span> <k=v>...` | Set key=value fields on a span |
| `log tree [--span <id>]` | Print indented span tree |
| `log list` | List all span UUIDs with name and status |
| `serve [name] [--file] [-t type] [--ttl] [--wrap]` | Pipe stdin (or `--file`) into daemon HTTP www; print URL |
| `web list` | List published temp files |
| `web url <name>` | Print URL for a published name |
| `web key-url <key>` | Print HTTP URL for a persistent key |
| `web rm <name>` | Remove a published temp file |
| `web status` | Print daemon HTTP base URL |
Global flags:
- `--home <path>` / `$UNIFIER_HOME` — top-level store directory
- `--chroot <name>` / `$UNIFIER_CHROOT` — scope data commands to `chroots/<name>/`
- `--no-daemon` — force direct filesystem access even when a hot daemon is running
- `--namespace <name>` / `$UNIFIER_NAMESPACE` — prefix keys for this invocation (overrides a bound namespace; empty disables it)
- `$UNIFIER_HTTP_PORT` — preferred localhost port for the daemon web server (default `17355`)
- `$UNIFIER_EVENT_TTL_SECS` — default event lifetime in seconds (default `86400` / 24h; `0` = never expire)
## Examples
```bash
# Build
cargo build --release
# Shared key-value
unifier put app/theme dark
unifier get app/theme
# Mailbox messaging
unifier send --from builder worker "rebuild docs"
unifier poll worker --ack
# Agent envelopes + wakeup (Jan listens on events.sock)
unifier message --from myagent youragent '{"hello":"world"}'
unifier event '{"name":"prices.updated"}' # expires in 24h
unifier event '{"name":"keep"}' --ttl 0 # never expires
unifier event '{"name":"job","ttl":60}' # payload ttl (seconds)
unifier daemon watch # prints {"kind":"mailbox","id":"...","from":"myagent","to":"youragent"}
# Cron drops
unifier cron 0_0_*_*_* "nightly backup"
unifier poll-cron --ack
# ACID ticks (daemon required; CLI uses .daemon/tick.sock)
unifier tick start swarm
unifier tick phase sense
unifier put board/x 1 # staged until tick end
unifier tick phase decide
unifier tick end
unifier daemon watch # also sees {"kind":"tick","tick":1,"phase":"sense",...}
# Chroots
unifier chroot init work
unifier --chroot work put deploy/target staging
unifier --chroot work get deploy/target # works
unifier get deploy/target # not found (global scope)
export UNIFIER_CHROOT=work
unifier send worker "task"
# Process-scoped key prefix (until the script exits or namespace set is called again)
unifier namespace set myscript
unifier put app/theme dark # keys/myscript/app/theme
unifier get app/theme # dark
unifier get /myscript/app/theme # same key, absolute (no extra prefix)
unifier --namespace other get k # keys/other/k for this call only
unifier namespace clear
# Managed SQLite
unifier sql create app
unifier sql exec app "CREATE TABLE items (id INTEGER PRIMARY KEY, name TEXT)"
unifier sql exec app "INSERT INTO items(name) VALUES ('alpha')"
unifier sql tables app --schema
unifier sql exec app "SELECT * FROM items"
unifier sql list
# Triples (auto-creates sqlite/triples.sqlite)
unifier triple add alice knows bob
unifier triple query --subject alice
unifier sql exec triples "SELECT * FROM triples"
# Span-tree logging
SPAN=$(unifier log start pipeline --field env=prod)
STEP=$(unifier log start fetch-prices --parent $SPAN)
unifier log event $STEP "fetched 42 rows" --field count=42
unifier log end $STEP
unifier log end $SPAN
unifier log tree # indented tree with events and fields
# Daemon web server (temp HTML reports)
unifier web list
unifier web url nightly
curl "$(unifier web url nightly)"
unifier web rm nightly
```
## Installation
```bash
cargo install unifier-cli
```
The binary is `unifier`. crates.io uses **`unifier-cli`** because `unifier` is already taken.
## Releasing
Same flow as jan-cli. From `unifier/`:
```bash
./scripts/release.sh # patch bump
./scripts/release.sh --minor
./scripts/release.sh --major
./scripts/release.sh --set 1.2.3
./scripts/release.sh --dry-run
./scripts/release.sh --resume
```
Monorepo aliases:
```bash
mdo release
mdo release-dry-run
mdo release-minor
mdo release-major
mdo publish
```
This bumps `Cargo.toml` + `CHANGELOG.md`, commits those version files, runs tests, then `cargo publish` as **unifier-cli**.
## Development
```bash
cargo test
cargo clippy -- -D warnings
cargo fmt
```
## Summary of changes
### Initial implementation
Created the `unifier` Rust CLI project with:
- **Filesystem-as-store model** — inspired by `filesystem-git-issues`: entities map to directories, scalar values to files, messages to UUID `.txt` drops.
- **State root** at `~/.local/unifier` with `$UNIFIER_HOME` / `--home` override.
- **Key-value store** — `put`, `get`, `del` under `keys/`, with atomic writes for `put`.
- **Mailbox** — `send`, `poll`, `ack` for point-to-point messages under `mailbox/<recipient>/`.
- **Cron postbox** — `cron`, `poll-cron`, `list` using five-field schedule directory names matched against the current clock.
- **Path safety** — keys and recipients reject `..`; message ack resolves paths relative to the store root.
- **Tests** — unit tests for cron parsing/matching; integration tests for put/get, mailbox, cron, and root.
- **`project.meta.yaml`** — monorepo metadata, build scripts, and roadmap for future applications (zshrc hooks, agent pipelines, job queues, etc.).
### Chroot support
Added isolated subtrees so multiple programs can share one global store without seeing each other's data:
- **On-disk layout** — `chroots/<name>/` contains a full copy of the layout (`keys/`, `mailbox/`, `cron/`).
- **`--chroot` / `$UNIFIER_CHROOT`** — all data commands operate only inside the named chroot; `UnifierHome::path()` returns the effective subtree root.
- **Admin commands** — `chroot init <name>` creates a new sandbox; `chroot list` enumerates existing chroots (always uses the global root).
- **Scope module** (`scope.rs`) — centralizes path validation: `resolve_under_root()` rejects `..` and absolute escapes; used by `list` and `ack`.
- **Isolation tests** — verify keys in one chroot are invisible to the global store and other chroots; path escape attempts fail.
- **Backward compatibility** — omitting `--chroot` preserves the original global-root behavior.
### Source layout
```
src/
main.rs # binary entry point
lib.rs # library + run()
home.rs # UnifierHome (global + effective root)
scope.rs # chroot path confinement
chroot.rs # chroot init/list
paths.rs # path builders for keys, mailbox, cron
postbox.rs # put/get/send/poll/ack operations
cron.rs # schedule parsing and matching
fs_text.rs # read/write helpers
constants.rs # directory name constants
error.rs # Error type
cli/
defs.rs # clap command definitions
mod.rs # dispatch
tests/
cli_integration.rs
```
### Hot daemon
Added an in-memory hot store with on-demand disk flush:
- **`HotStore`** (`store.rs`) — loads keys/mailbox/cron from disk; tracks dirty entries; `flush()` writes only changed data
- **Daemon process** — `unifier daemon start|run|stop|status|flush|watch`; control socket at `<store>/.daemon/unifier.sock`; event output at `<store>/.daemon/events.sock`
- **Mailbox envelopes** — messages store `{id, from, to, payload}`; send/message/event broadcast a wakeup notice (`to` + `id`) so Jan can fetch `mailbox/<to>/<id>.txt`
- **Auto-routing** — data commands use the daemon when running; `--no-daemon` forces direct filesystem access
- **Idle auto-flush** — after 5s with no daemon requests, dirty state is written to disk if 1-minute loadavg ≤ 0.2 (`UNIFIER_IDLE_FLUSH_SECS`, `UNIFIER_IDLE_MAX_LOAD`; set flush secs to 0 to disable). Skips an active tick.
- **Idle auto-stop** — after 10 minutes with no control/event activity, no active tick, and no event subscribers, the daemon flushes and exits (`UNIFIER_IDLE_STOP_SECS`; set to 0 to disable). Prevents Gradle-style auto-started daemons from living forever when unused.
- **Missing-home exit** — if the store root directory disappears (e.g. a test `TempDir` is dropped), the daemon exits immediately so orphans do not pile up under `/tmp`.
- **`unifier daemon gc`** — kill orphan `unifier daemon run` processes whose `--home` path no longer exists (`--dry-run` to list only).
### Event expiry
Posted events are stored as `{payload, created_at, expires_at}` under `events/<uuid>.json` so they do not accumulate forever:
- Default TTL is **24 hours** (`$UNIFIER_EVENT_TTL_SECS` to change the default; `0` = never)
- `unifier event '...' --ttl <secs>` overrides (`--ttl 0` never expires)
- Payload may set `"ttl": 60` (seconds) or `"expires_at": "<rfc3339>"` (`null` = never)
- Expired files are deleted on daemon load, flush, and a 30s sweep
### Process namespaces
`namespace set` writes a binding for the calling process under `.daemon/namespaces/<pid>`. Later key operations from that process or a descendant prefix `put`/`get`/`del` and tick lock names with `<namespace>/`. Bindings are dropped when the process exits (reaped) or `namespace clear` / a new `namespace set` runs. Keys starting with `/` skip the prefix. `$UNIFIER_OWNER_PID` overrides which process owns the binding (the CLI parent, by default).
### Span-tree logging
Added `unifier log` for tracing Jan cron scripts and agent pipelines:
- **`log start <name> [--parent <id>] [-f k=v]`** — create a new span under `logs/<uuid>.json`; prints UUID for capture
- **`log end <id>`** — record `ended_at` on a span
- **`log event <span> <msg> [-f k=v]`** — append a timestamped event to a span's `events` array
- **`log field <span> k=v...`** — set or update key/value fields on a span
- **`log tree [--span <id>]`** — render the full span tree (or subtree) as indented text with events and fields
- **`log list`** — list all spans with UUID, name, and open/ended status
- Spans load directly from disk (no daemon round-trip); sorted by `started_at` for stable tree output
### Daemon web server
Temp static files for Jan report export and quick HTML previews, plus live key reads for deep links from those pages:
- **`serve [name]`** — read stdin (or `--file`), write under `.daemon/www/`, print `http://127.0.0.1:<port>/<name>`
- **`--wrap`** — embed body in Unifier HTML chrome (brand header + “all reports” link)
- **`--ttl <secs>`** — auto-expire published files; GC on HTTP access / periodic sweep
- **`web list|url|rm|status`** — manage and inspect published files
- **`web key-url <key>`** — print `http://127.0.0.1:<port>/keys/<key>`
- **`GET /keys/<key>`** — return the live key value (`application/json` or `text/plain`); browsers get an HTML wrap when `Accept` prefers `text/html`
- **`GET /keys/`** / **`GET /keys/<prefix>/`** — list present keys (plain lines or HTML)
- Daemon binds localhost (preferred port **17355**, or `$UNIFIER_HTTP_PORT`, else ephemeral); port stored in `.daemon/http.port`
- HTTP activity refreshes the daemon idle timer so open reports are not cut off mid-view
### Daemon durability
Three fixes that agents launched by a scheduler depend on:
- **One connection, many requests** — the daemon serves a connection until the peer closes it (or goes quiet for 2s). It used to close after the first response, which broke `poll --ack`: the ack hit a broken pipe and the messages stayed in the mailbox, so the next wake reprocessed them.
- **Own session** — a spawned daemon calls `setsid`, so it survives the shell, cron job, or agent that auto-started it rather than taking a SIGHUP with it.
- **Flush on signal** — `SIGTERM` / `SIGINT` / `SIGHUP` drain hot state to disk before exit. Previously anything not yet flushed was lost.
### Multi-agent example
The job pipeline lives in the personal `dotfiles` jan tree
(`dotfiles/jan/jobs.yaml`, sibling of this Incubator checkout): four agents that
only ever talk through Unifier.
```
job-scout --(message)--> job-analyst --(message)--> job-responder --(event)--> job-digest
```
`job-scout` runs on cron and mails new job postings; `job-analyst` and `job-responder` have no schedule at all and are woken by Jan when mail arrives for their leaf name; `job-digest` renders the result with `serve`. Each agent drains its whole mailbox per wake, so coalesced or duplicated wakeups are harmless. Manual poke: `jan jobs interact run --to job-analyst --message '…'`.
### Managed SQLite
Databases live under `sqlite/<name>.sqlite` (scoped by `--chroot` when set). `sql list` / `sql tables` / `sql exec` manage and query them directly on disk (no hot-daemon round-trip). The reserved `triples` database stores unique subject/predicate/object rows; use `triple add` / `triple query` or `sql exec triples '...'`.
### Source layout (updated)
```
src/
store.rs # HotStore (in-memory + flush)
daemon/ # protocol, client, server, lifecycle
...
```
## Roadmap
See `project.meta.yaml` for the full list. Next major themes:
- Tick-based agents (sense/decide/act lockstep, concurrency modes, differential sensing)
- Factorio factory model: isolated arms coordinated toward a shared system goal