# rSearch
FIPS-compliant, cluster-ready log search server written in Rust. A lightweight
replacement for the Graylog/OpenSearch stack: single static binary, no JVM,
no garbage collector.
## Highlights
- **OpenSearch-compatible wire subset** — `_bulk`, `_search`, `_cat`, and
cluster health endpoints; works unmodified with Vector, Fluent Bit,
Filebeat, and Grafana's Elasticsearch datasource
- **FIPS from the ground up** — all TLS via rustls with the aws-lc-rs FIPS
provider (CMVP-validated), approved algorithms only, enforced by a
`cargo deny` gate in CI
- **Cluster-ready by design** — immutable index splits in object storage
(S3, MinIO, or local disk), Postgres metastore/control plane, stateless
searchers; a single node is just a cluster of one
- **HA on plain block storage** — the `replicated` backend keeps split
copies on N nodes' local disks with quorum writes, peer reads, and
automatic re-replication; no external object store required
- **Tantivy index engine** — Lucene-class full-text search and
ES-compatible aggregations with predictable, GC-free latency
## FIPS compliance
All cryptography flows through **aws-lc-rs in FIPS mode** (the AWS-LC
Cryptographic Module, CMVP-validated under FIPS 140-3), pulled in via
rustls's `fips` feature. The exact CMVP certificate is determined by the
pinned `aws-lc-fips-sys` version in `Cargo.lock`; see the
[aws-lc-rs FIPS documentation](https://aws.github.io/aws-lc-rs/fips.html)
for the certificate covering each module release. The server refuses to
start TLS unless `ServerConfig::fips()` reports true.
Enforcement is structural, not procedural:
- `deny.toml` bans `ring`, `openssl`, `md5`/`md-5`, `argon2`, `bcrypt`,
`scrypt`, and unvalidated RustCrypto implementations from the entire
dependency graph
- `scripts/ci.sh` runs `cargo deny check bans licenses` on every build;
a banned crate anywhere in the tree fails CI
- Building the FIPS module from source requires CMake and Go on the
build host (validated-module build procedure)
Passwords (phase 9) use PBKDF2-HMAC-SHA256. Non-security hashing
(document IDs, cache keys) uses clearly non-cryptographic hashes so
intent stays auditable.
Documented `deny.toml` exceptions (both wire-protocol legacy, not
security functions): S3 Content-MD5 integrity headers
(aws-smithy-checksums), and the unused Postgres legacy md5 auth path in
sqlx — **deployments must configure Postgres with `scram-sha-256`
authentication** (the default since Postgres 14).
## Status
v1 core complete. See `working-plan.md` for the phase breakdown and
`BENCHMARKS.md` for the rSearch-vs-OpenSearch gate results (~9× less
memory, ~5× less CPU at 5k events/s, query-latency parity).
## Architecture (short version)
Ingest nodes accept logs (`_bulk`, syslog-TLS, GELF), write a local WAL,
build immutable Tantivy splits, and publish them to object storage. The
Postgres metastore tracks splits, streams, nodes, and retention. Stateless
search nodes prune splits by time range via the metastore and execute
queries directly against storage through a local cache. One binary runs any
combination of roles: `rsearch --roles ingest,search,control`.
## Install
From crates.io (builds the FIPS module from source — needs CMake, Go,
and clang on the build host):
```bash
CC=clang CXX=clang++ cargo install rsearch-server # installs the `rsearch` binary
```
Or build from a checkout with `CC=clang CXX=clang++ cargo build
--release` (clang is required: the aws-lc FIPS delocator rejects newer
GCC assembly). Container images build from the included `Dockerfile`.
## Quick start
```bash
# 1. dependencies (Postgres 14+ with scram-sha-256, object storage or local disk)
docker compose up -d postgres minio # or bring your own
# 2. build (see Install above for toolchain requirements)
CC=clang CXX=clang++ cargo build --release
# 3. run a single-node cluster (all roles)
export DATABASE_URL=postgres://rsearch:rsearch@localhost:5432/rsearch
./target/release/rsearch --roles ingest,search,control
```
Migrations run automatically at startup. On first boot, auth is in
**bootstrap mode** (a startup warning is logged); create the first admin
to arm enforcement cluster-wide:
```bash
curl -XPUT localhost:9200/_rsearch/users/admin \
-d '{"password":"choose-a-strong-one"}'
```
A reference multi-node topology (2 ingest + 2 search + 1 control over
Postgres and MinIO) is in `docker-compose.yml`; the kill-a-node test
suite that exercises it is `tests/cluster/run-cluster-test.sh`.
## HA on block storage (no object store)
The `replicated` storage backend turns each node's local disk (any block
device — EBS, iSCSI, plain SATA) into cluster storage with no external
object store: only rSearch and Postgres run. Every split is written to
`storage.replication_factor` nodes (quorum-acknowledged before the
ingest WAL lets go of the data), reads fall back to a live holder over
the internal peer API, and the control leader re-replicates the copies
of a node that goes silent for `control.repair_stale_secs` (default
5 minutes). New or empty nodes absorb new writes first, so capacity
rebalances as data churns through retention.
```toml
[node]
advertise_addr = "node1.internal:9200" # peers must be able to dial this
[storage]
backend = "replicated"
root = "/var/lib/rsearch/objects" # this node's local object root
replication_factor = 2
[cluster]
internal_token = "<openssl rand -hex 32, same on every node>"
```
Operational notes:
- Scale in gracefully with `POST /_rsearch/nodes/{id}/drain`: the node
keeps serving reads while the leader copies its objects to the other
nodes, and it refuses new `_bulk` traffic (503) so its WAL empties out
(repoint syslog/GELF shippers yourself). When the leader logs "drain
complete" (its `object_locations` rows are gone) the node can be shut
down. `DELETE` on the same path cancels a drain. A draining node takes
no writes or repair copies, so don't leave the flag set on a node you
mean to keep: `GET /_cat/nodes` reports `draining_since_secs`, and the
leader warns every tick once a drain outlives
`control.drain_warn_secs` (default 1h). If under-replicated data has
nowhere else to go, repair will fall back to a draining node rather
than leave a key one failure from loss (the drain job moves the copy
off again later).
- Postgres holds placement and all metadata — run it HA too, or it is
the single point of failure.
- With factor 2, the window between a node dying and repair completing
is one further failure away from data loss; size
`repair_stale_secs` and node count accordingly.
- A node returning after its registry entry expired may hold orphaned
object files (repair already replaced its copies); these are inert
and can be cleaned by wiping the object root before restart.
- The ingest WAL stays node-local: docs acked but not yet published when
a node dies are recovered by WAL replay when that node (or its volume)
returns — same recovery story as the fs backend.
- TLS between peers uses the FIPS provider; set `cluster.peer_ca_file`
to a PEM bundle when node certificates are signed by an internal CA
(otherwise the public webpki roots apply). Peer endpoints share the
API listener — keep the port on a trusted network segment.
- The server refuses to start the replicated backend with a wildcard
(`0.0.0.0`/`[::]`) advertise address — peers must be able to dial
`node.advertise_addr`.
A containerized reference topology (3 nodes, per-node volumes standing
in for block devices) is in `docker-compose-replicated.yml`. Two test
suites exercise the backend: `tests/cluster/run-replicated-test.sh`
(process-level: kill-a-holder, repair, drain, fan-out GC) and
`tests/cluster/run-ha-compose-test.sh` (container-level: data-node
death, leader failover, volume-reattach rejoin).
## Configuration
Config loads from an optional TOML file (`--config`) with `RSEARCH_`
environment overrides (nested keys use `__`, e.g.
`RSEARCH_HTTP__TLS__ENABLED=true`). See `rsearch.example.toml` for the
full annotated set. Storage backends — S3, S3-compatible (MinIO), and
local filesystem — are equal citizens; self-hosted/air-gapped
deployments use static credentials in config so the AWS credential chain
and IMDS are never touched.
## API surface (OpenSearch-compatible subset)
| Ingest | `POST /_bulk`, `POST /{index}/_bulk` |
| Search | `POST /{index}/_search`, `POST /_msearch`, `GET /{index}/_mapping` |
| Index admin | `PUT /{index}` (mapping), `GET /_cat/indices` |
| Cluster | `GET /`, `GET /_cluster/health`, `GET /_cat/nodes` |
| Streams | `PUT /_rsearch/streams/{name}/retention`, routing rules under `/_rsearch/routing_rules` |
| Alerts | `PUT/GET/DELETE /_rsearch/alerts[/{name}]` (scheduled query → webhook) |
| Auth | `POST /_rsearch/login`, users/api_keys under `/_rsearch/` |
| Observability | `GET /metrics` (Prometheus), `GET /_rsearch/stats` (JSON) |
Query DSL subset: `match_all`, `bool`, `term`, `terms`, `range`,
`exists`, `match`, `match_phrase`, `query_string`. Aggregations pass
through Tantivy's ES-compatible module (terms, date_histogram, stats,
percentiles, cardinality, …).
Inputs beyond HTTP: syslog (RFC 5424 + 3164, UDP/TCP, optional TLS) and
GELF (TCP), each routable to a stream and subject to routing rules.
### Loki-compatible API (Grafana Logs Drilldown)
rSearch also speaks a subset of Loki's HTTP query API, so Grafana's
built-in **Loki datasource** — and with it **Logs Drilldown** — works
with no plugins: point a Loki datasource at the rSearch URL (Basic auth
or an API key as a bearer credential) and browse.
| `GET/POST /loki/api/v1/query_range` | log selectors → `streams`, metric queries → `matrix` |
| `GET/POST /loki/api/v1/query` | instant queries |
| `GET /loki/api/v1/labels`, `/label/{name}/values` | label discovery |
| `GET/POST /loki/api/v1/series` | series matching |
| `GET/POST /loki/api/v1/index/volume`, `/index/volume_range` | Drilldown's volume breakdowns |
| `GET /loki/api/v1/tail` | WebSocket live tail (poll-backed) |
| `GET /ready` | Loki readiness probe (open, like `/health`) |
Model mapping: the `service_name` label is the stream name (every stream
appears as a browsable service); other labels are the stream's
keyword-mapped fields, with values served from terms aggregations
(capped at 1000). A log line is the doc's `message` field, or the raw
`_source` JSON when there is none.
LogQL coverage is the subset Grafana sends: selectors with
`=`, `!=`, `=~`, `!~`; line filters `|=`, `!=`, `|~`, `!~`;
`count_over_time` and `rate` (correct sliding-window math for any
step/range combination), optionally wrapped in `sum` / `sum by (label)`
— one grouping label. Like `/_msearch`, the Loki surface requires
search-level auth with global stream access, and selectors must contain
at least one matcher that doesn't match the empty string.
Line filters are true substring/regex tests against the rendered line —
never a tokenized index match that could miss "error" when searching
"err". The trade-off: a filtered query (or filtered metric) examines up
to 5000 selector-matching docs per stream, newest first, and sets a
response warning when that scan window saturates. Per-stream failures in
multi-stream queries degrade to `warnings` with partial results instead
of failing the whole query. Tails are capped at 16 concurrent sessions
and 1 hour per session, with WebSocket pings reaping dead peers.
`GET /metrics` serves Prometheus text format: ingest throughput and
queue depth, WAL backlog (`rsearch_wal_outstanding_records` — watch this
for restart-replay memory pressure), cluster node liveness/draining
gauges, and on control nodes leadership plus repair/drain activity. It
requires search-level auth like `/_rsearch/stats`; point a scrape job at
it with an API key:
```yaml
scrape_configs:
- job_name: rsearch
metrics_path: /metrics
authorization:
type: Bearer
credentials: <api key from POST /_rsearch/api_keys>
static_configs:
- targets: ["node-a:9200", "node-b:9200", "node-c:9200"]
```
## Workspace crates
| Crate | What it is |
|-------|------------|
| `rsearch-server` | The `rsearch` binary: HTTP API, roles, control plane |
| `rsearch-common` | Config, roles, FIPS TLS, crypto helpers |
| `rsearch-storage` | Storage backends: local fs, S3/MinIO, node-replicated |
| `rsearch-index` | ES-style mappings on Tantivy; immutable split files |
| `rsearch-metastore` | Postgres metastore: streams, splits, placement, leadership (migrations embedded) |
| `rsearch-ingest` | `_bulk`/syslog/GELF parsing, WAL, indexer pipeline |
| `rsearch-search` | Query-DSL subset executed over published splits |
Release procedure (versioning, publish order) is in `RELEASING.md`.
## Web console
`ui/` is a Next.js app (search, streams/retention, alerts, users & API
keys). `cd ui && npm install && npm run build`. The API base is resolved
at runtime: replace the served `/env.js` (bind-mount, S3 object
overwrite, or a container entrypoint writing it) with
```js
window.__RSEARCH_API__ = "https://rsearch.example.com:9200"; // "" = same origin
```
to point an already-built console at any cluster — no rebuild. When
`/env.js` sets nothing, the `NEXT_PUBLIC_RSEARCH_API` build-time value
applies (default `http://localhost:9200`).
## License
rSearch is free software, licensed under the [GNU General Public
License v3.0 or later](LICENSE).