oxen-server
The server for remote oxen repositories.
Remote repositories have the same internal structure as local ones, with the caveat that all the data is in the .oxen/ dir and not duplicated into a "local workspace".
Notable configuration sections:
- Prometheus Metrics
- OpenTelemetry Tracing
- FmtSpan Events
- Stacking Tracing Layers | Writing Spans to Logs & OTel
Build
See the prerequisites section of the main readme before developing.
Use the standard cargo ... --workspace commands and cargo ... -p oxen-server commands.
Run
To run a local Oxen Server, generate a config file and token to authenticate the user:
Copy the config to the default locations:
Set where you want the data to be synced to.
The default sync directory is ./data/.
To change, set the SYNC_DIR environment variable to a path:
You can also create a .env.local file in the crates/server/ directory which can contain the SYNC_DIR variable to avoid setting it every time you run the server.
Run the server:
Or run the compiled binary directly:
To run the server with live reload, use bacon:
Then run the server like this:
API Examples
Server defaults to localhost 3000.
You can grab your auth token from the config file above (~/.oxen/user_config.toml):
List Repositories
Create Repository
Logging
Oxen uses structured logging. It outputs to STDERR by default but can be configured with rotating log files. See Logging for details.
By default, oxen-server logs at the WARN level. Set RUST_LOG to change.
Prometheus Metrics
oxen-server exposes a Prometheus-compatible
metrics endpoint. This allows you to monitor server health, track request
counts, error rates, and other operational metrics using standard Prometheus
tooling.
Compile-time feature flag
Metrics collection requires the metrics Cargo feature. Without it, all
metric collections (counter!, histogram!, etc.) compile to no-ops —
no counters are recorded and no /metrics endpoint is served,
regardless of environment variables.
The metrics feature is included in production, so a production build
already has it:
To enable metrics alone (without OpenTelemetry tracing or other production features):
# just metrics, for any crate
# or per-crate
If OXEN_METRICS_PORT is set at runtime (to a value other than off)
but the binary was compiled without the metrics feature, the server
logs an error at startup explaining the mismatch.
How it works
On startup (when compiled with metrics), oxen-server launches a
lightweight HTTP server (separate from the main API) that serves metrics
in the Prometheus exposition format. Any counters, gauges, or histograms
recorded via the metrics crate are
automatically exposed.
Configuration
The metrics endpoint is opt-in. Set OXEN_METRICS_PORT to a port number
to enable it.
| Variable | Description | Default |
|---|---|---|
OXEN_METRICS_PORT |
Port for the metrics HTTP server (opt-in) | (none — disabled) |
OXEN_METRICS_PORT=off |
Explicitly disable the metrics endpoint | -- |
# No metrics server (default)
# Enable metrics on port 9090
OXEN_METRICS_PORT=9090
# Enable metrics on a custom port
OXEN_METRICS_PORT=9100
# Explicitly disable metrics
OXEN_METRICS_PORT=off
Verifying with curl
This returns all registered metrics in Prometheus text format, e.g.:
# TYPE oxen_errors_total counter
oxen_errors_total{module="commits",error="not_found"} 3
Integrating with Prometheus
Add a scrape target to your prometheus.yml:
scrape_configs:
- job_name: oxen-server
scrape_interval: 15s
static_configs:
- targets:
If you run multiple oxen-server instances, list each one (or use service
discovery):
scrape_configs:
- job_name: oxen-server
static_configs:
- targets:
- "oxen-1.internal:9090"
- "oxen-2.internal:9090"
Integrating with Grafana
Once Prometheus is scraping the endpoint, add it as a data source in Grafana and build dashboards using PromQL queries. For example:
rate(oxen_errors_total[5m])
OpenTelemetry Tracing
oxen-server can export tracing spans to any OTLP-compatible collector
(Jaeger, Tempo, Honeycomb, Datadog, etc.). This requires building with the
otel feature flag:
At runtime, set OXEN_OTEL_ENDPOINT to enable export:
# gRPC (default protocol)
OXEN_OTEL_ENDPOINT=localhost:4317
# HTTP/JSON
OXEN_OTEL_ENDPOINT=localhost:4318 OXEN_OTEL_PROTOCOL=http
# Or include the protocol in the endpoint string directly
OXEN_OTEL_ENDPOINT=http://localhost:4318
OXEN_OTEL_ENDPOINT=grpc://localhost:4318
| Variable | Description | Default |
|---|---|---|
OXEN_OTEL_ENDPOINT |
Collector endpoint URL. Absent = disabled. | (none) |
OXEN_OTEL_PROTOCOL |
Transport: grpc or http |
grpc or whatever is listed in OXEN_OTEL_ENDPOINT |
The standard OTEL_EXPORTER_OTLP_ENDPOINT variable is also respected as a
fallback if OXEN_OTEL_ENDPOINT is not set.
When the otel feature is not compiled in, no OpenTelemetry dependencies are
included and the env vars are ignored.
RUST_LOG and span visibility
The RUST_LOG filter is global — it gates what reaches all tracing
outputs, including the OpenTelemetry exporter. #[tracing::instrument]
creates spans at INFO level by default. The server defaults to
LevelFilter::WARN when RUST_LOG is not set, which means all
#[instrument] spans are silently dropped before the OTel layer sees
them.
To get full traces in Jaeger (or any collector), explicitly set
RUST_LOG=info:
OXEN_OTEL_ENDPOINT=http://localhost:4317 RUST_LOG=info
Without RUST_LOG=info, the OTel exporter is active but receives no spans.
The TracingLogger HTTP root span is also at INFO level, so it is
similarly affected.
For targeted verbosity (e.g. keep third-party crates quiet), use a filter directive:
RUST_LOG="warn,liboxen=info,oxen_server=info,tracing_actix_web=info"
Quick Start with Jaeger
# Start Jaeger all-in-one: https://www.jaegertracing.io/docs/2.17/
# Start oxen-server with OTel export
OXEN_OTEL_ENDPOINT=http://localhost:4317 RUST_LOG=info
# View traces at http://localhost:16686 under service "oxen-server"
FmtSpan Events
Span lifecycle events (creation, entry, exit, close) can be emitted as
additional log lines on stderr. This is useful for seeing timing of
#[instrument]-annotated functions without a full tracing collector.
Set OXEN_FMT_SPAN to enable:
# Log when spans close (includes elapsed time)
OXEN_FMT_SPAN=CLOSE
# Log all span lifecycle events
OXEN_FMT_SPAN=FULL
# Combine specific events
OXEN_FMT_SPAN="NEW|CLOSE"
Accepted values: NEW, CLOSE, ENTER, EXIT, ACTIVE (enter+exit),
FULL (all), NONE, 1/true (alias for CLOSE).
No feature flag or additional dependencies are required.
Stacking Tracing Layers
All tracing outputs can be enabled simultaneously. For example, to get stderr output with span timing, JSON file logs, and OpenTelemetry export:
OXEN_LOG_DIR='/var/log/oxen' \
OXEN_FMT_SPAN='CLOSE' \
OXEN_OTEL_ENDPOINT='http://localhost:4317' \
RUST_LOG='info' \