durable-streams-rust-server

Published crate: durable-streams-server. Repository: durable-streams-rust-server.

Rust implementation of the durable streams protocol, with idiomatic approaches to fulfilling the protocol specification and feature parity with the caddy implementation. This project exists to validate ai-enabled development approaches, to pressure test the durable streams approach and to document how to deliver an end to end solution with durable streams, electric sql and postgres, using the gatekeeper auth pattern.

Documents https://thesampaton.github.io/durable-streams-rust-server/

Quick start

cargo run

The server listens on http://localhost:4437 with streams at /v1/stream/.

Storage backends

The default storage mode is in-memory. For persistence, choose a backend via DS_STORAGE__MODE:

# Run with durable file storage
DS_STORAGE__MODE=file-durable DS_STORAGE__DATA_DIR=./data cargo run

# Run with crash-resilient acid storage
DS_STORAGE__MODE=acid DS_STORAGE__DATA_DIR=./data cargo run

Build and test

cargo build          # compile
cargo test           # unit + integration tests
cargo clippy -- -D warnings  # lint
cargo fmt            # format

Performance builds (PGO)

Use profile-guided optimization (PGO) for production-oriented builds:

# 1) Generate and merge fresh profiles using benchmark traffic
make pgo-train

# 2) Build release binary with profile-use (fails if profile is missing/stale)
make release-pgo

# 3) Optional: compare profile-use performance vs baseline
make pgo-benchmark

Key PGO controls:

• PGO_DIR (default: /tmp/ds-pgo) for profile artifacts
• PGO_REQUIRE_FRESH (default: true) to enforce freshness checks
• PGO_MAX_AGE_HOURS (default: 168) maximum allowed profile age
• PGO_WARN_MISSING (default: false) enables/disables LLVM missing-profile warnings

CI also has a dedicated PGO workflow at .github/workflows/pgo-release.yml that trains profiles and publishes a release-pgo artifact.

Benchmark findings

The latest benchmark matrix findings are in docs/benchmark-report.md. It includes:

• 3 runs each for Rust (memory, file-durable, acid), Node (memory, file), and Caddy plugin (memory, acid label for plugin file-backed mode)
• fresh Rust PGO profile training before measurements
• explicit durability semantics notes for acid vs Caddy acid interpretation

Conformance

This implementation targets full conformance with @durable-streams/server-conformance-tests@0.2.3 against spec commit a347312.

Run conformance tests locally:

DS_SERVER__LONG_POLL_TIMEOUT_SECS=2 DS_SERVER__SSE_RECONNECT_INTERVAL_SECS=5 cargo run &

cd /tmp/conformance-run
npm init -y
npm install @durable-streams/server-conformance-tests@0.2.3
cat > conformance.test.mjs << 'EOF'
import { runConformanceTests } from "@durable-streams/server-conformance-tests";
runConformanceTests({ baseUrl: process.env.CONFORMANCE_TEST_URL, longPollTimeoutMs: 2000 });
EOF

CONFORMANCE_TEST_URL=http://localhost:4437 npx vitest run conformance.test.mjs

Configuration

The server now supports layered TOML config with env overrides.

Load order (later wins):

1. built-in defaults
2. config/default.toml (if present)
3. config/<profile>.toml (if present, via --profile)
4. config/local.toml (if present; gitignored)
5. extra file from --config <path>
6. environment variables

CLI:

# Uses config/default.toml + config/dev.toml (+ config/local.toml if present)
cargo run -- --profile dev

# Add an extra override file on top
cargo run -- --profile prod --config /etc/durable-streams/override.toml

Environment variables use DS_ prefix with double-underscore section separators: