# Power-House Operations Guide
Release scope: Power House v0.3.5.
This guide documents the retained boot-node, JULIAN network, and blob-service
operations in v0.3.5. For the current three-validator native RPC topology and
public HTTPS edge, use
[Production RPC Deployment](production_rpc_deployment.md). Commands here
assume explicit operator control so each change remains auditable.
The production public service is named **LAX MFENX RPC**.
## 1. Requirements
- Rust toolchain (via `rustup`) and `cargo` on the build machine.
- `ssh`/`scp` access to every VPS (keys or password).
- systemd on the VPS (recommended).
- Python 3 on the VPS (healthcheck runner).
## 2. Build the binary
```bash
cargo build --release --features net --bin julian
```
## 3. Install/upgrade on each VPS
```bash
scp target/release/julian root@HOST:/usr/local/bin/julian.new
ssh root@HOST 'install -m 0755 /usr/local/bin/julian.new /usr/local/bin/julian && rm /usr/local/bin/julian.new'
```
## 4. Systemd units and environment
Use the unit files in `infra/systemd/` as the canonical service definitions.
Deploy them with:
```bash
scripts/deploy_units.sh root@BOOT1 root@BOOT2
```
Environment files (store in `/etc/powerhouse/`):
- `powerhouse-common.env` (shared settings)
- `powerhouse-boot1.env` (boot1 node)
- `powerhouse-boot2.env` (boot2 node)
Copy the examples from `infra/systemd/` and fill in real values.
The node wrappers read these files and start `/usr/local/bin/julian` using
`/usr/local/bin/powerhouse-boot.sh`.
The deployment script installs refreshed `.env.example` files but refuses to
start a node until the two live environment files already exist. This prevents
an upgrade from silently replacing identities, peer addresses, or secrets.
Recommended production flags:
- `--blob-dir /var/lib/powerhouse/<node>/blobs`
- `--blob-listen 0.0.0.0:8181`
- `--max-blob-bytes <bytes>`
- `--blob-auth-token <token>` (enforces Authorization or x-api-key)
- `--blob-max-concurrency <n>` (defaults to 128)
- `--blob-request-timeout-ms <ms>` (defaults to 10000)
- `--attestation-quorum <n>`
- `--metrics :9100` (Prometheus metrics)
## 5. Health checks + alerts
Timers are included for continuous health checks:
```bash
systemctl enable --now powerhouse-healthcheck@boot1.timer
systemctl enable --now powerhouse-healthcheck@boot2.timer
```
The healthcheck verifies:
- systemd service state
- blob `/healthz`
- metrics `/metrics`
- finality progress (stall detection)
Alerts are delivered via `/usr/local/lib/powerhouse/alert.sh`.
If `PH_ALERT_EMAIL` is empty, alerts log to syslog only. To enable email,
set `PH_ALERT_EMAIL`/`PH_ALERT_FROM` and wire any local MTA (msmtp, postfix,
or equivalent). Keep SMTP credentials in a root‑only file.
## 6. Metrics + log export
Metrics endpoint:
```bash
curl http://<host>:9100/metrics
```
Hourly log exports (for shipping/archival):
```bash
systemctl enable --now powerhouse-log-export@boot1.timer
systemctl enable --now powerhouse-log-export@boot2.timer
```
Exports are written to `PH_LOG_EXPORT_DIR` (gzip). Optional shipping can be
enabled by setting `PH_LOG_SHIP_HOST` and related vars in the node env file.
## 7. Metrics snapshots (soak test)
Enable metrics snapshots (every 5 minutes):
```bash
systemctl enable --now powerhouse-metrics@boot1.timer
systemctl enable --now powerhouse-metrics@boot2.timer
```
Snapshots append to `PH_METRICS_OUT_DIR/metrics-<node>.jsonl`. Use this log to
baseline 24‑hour soak behavior (finality rate, rejects, invalid envelopes).
## 8. Backups + restore
Daily backups are enabled by:
```bash
systemctl enable --now powerhouse-backup@boot1.timer
systemctl enable --now powerhouse-backup@boot2.timer
```
Manual backup:
```bash
/usr/local/lib/powerhouse/backup.sh
```
Restore:
```bash
/usr/local/lib/powerhouse/restore.sh /var/backups/powerhouse/<archive>.tar.zst
```
## 9. Rollback
Use versioned releases under `/opt/powerhouse/releases`:
```bash
/usr/local/lib/powerhouse/deploy_release.sh /path/to/julian
/usr/local/lib/powerhouse/rollback.sh
```
## 10. Blob/DA endpoints
- Health: `curl http://<host>:8181/healthz`
- Submit: `curl -X POST http://<host>:8181/submit_blob -H 'X-Namespace: default' -H 'X-Fee: 0' --data-binary @file.bin`
- Commitment: `curl http://<host>:8181/commitment/default/<hash>`
- Sample: `curl "http://<host>:8181/sample/default/<hash>?count=2"`
- Prove storage: `curl http://<host>:8181/prove_storage/default/<hash>/0`
- Rollup settle: `curl -X POST http://<host>:8181/rollup_settle -H 'Content-Type: application/json' -d '{"namespace":"default","share_root":"…","payer_pk":"…","fee":1000,"mode":"optimistic"}'`
If `--blob-auth-token` is set, add:
- `Authorization: Bearer <token>` or `x-api-key: <token>`
Nonzero fees require a funded `X-Publisher` account and its matching
`X-Publisher-Sig`; use the stake CLI to provision balances before testing fees.
## 10.1 JSON-RPC publication gate
Enable native-chain finality on every validator with the same chain ID,
membership policy, quorum, and initial stake registry. Only RPC-facing nodes
need `--evm-rpc-listen`; validators without an HTTP endpoint still require
`--evm-chain-id`.
The first start writes `native_chain_state.json` under `--blob-dir`, or under
`--log-dir` when no blob directory is configured. Genesis commits the validator
set, strict-majority quorum, and initial balances. Fund the registry before the
first start. Later configuration mismatches fail startup instead of creating a
different chain.
Validate the complete three-validator transaction path before deployment:
```bash
scripts/test_native_rpc_cluster.sh
```
Before publishing an EVM-compatible endpoint in ChainList, run:
```bash
python3 scripts/check_rpc.py https://rpc.example.org --expected-chain-id 177155 --require-cors
```
The probe requires working DNS/TLS, consistent `eth_chainId` and
`net_version`, a valid latest block response, and browser CORS when requested.
It is an endpoint integrity check, not a substitute for consensus-state audits.
See [RPC Operations](rpc_operations.md).
## 10.2 External DA publisher (optional)
Power-House can push DA commitments to an external API (Celestia/Ethereum-compatible
gateway, or a custom relay). Configure via environment:
- `PH_DA_ENDPOINT` (required) — URL to POST commitment JSON
- `PH_DA_PROVIDER` (default: `generic`)
- `PH_DA_AUTH_TOKEN` (optional bearer token)
- `PH_DA_PUBLISH_INTERVAL_MS` (default: `5000`)
- `PH_DA_PUBLISH_INLINE` (default: `false`) — attempt publish inline on submit
- `PH_DA_RETRY_BACKOFF_MS` (default: `60000`)
- `PH_DA_PRUNE_AFTER_PUBLISH` (default: `false`) — removes `.blob` + `shares/` after publish
Outbox file: `da_outbox.jsonl` under the blob dir. Successful publishes append to
`da_published.jsonl`. The `/commitment` endpoint returns `da_receipt` when present.
## 11. Namespace policies and rate limits
`blob_policy.json` supports per-namespace guards:
- `max_bytes` (size cap)
- `min_fee` (fee floor)
- `max_per_min` (rate limit per namespace)
These are enforced at ingest time and are the first line of defense against
abuse on public endpoints.