Crate hyperi_rustlib 

hyperi-rustlib

There’s plenty of sage advice out there about how to run Rust services in production at scale – config cascades, structured logging, masking secrets, multi-backend secrets management, Prometheus, OpenTelemetry, Kafka transports, tiered disk-spillover sinks, adaptive worker pools, graceful shutdown – but almost none of it as code you can just install and use.

This is that code.

Opinionated, drop-in, working out of the box. The patterns from blog posts, watercooler chats and beers with your Google mates as actual library – not a framework you assemble from twenty crates and 8 weeks of munging.

Built as the foundation for HyperI’s PB/hr data services. Generic enough that you don’t need to be at HyperI to use it.

Full reference docs live under docs/. Start at docs/README.md for the entry-point index.

Quick Start

use hyperi_rustlib::{env, config, logger, metrics};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Detect runtime environment
    let environment = env::Environment::detect();
    println!("Running in: {:?}", environment);

    // Initialise logger (respects LOG_LEVEL env var)
    logger::setup_default()?;

    // Load configuration with 7-layer cascade
    config::setup(config::ConfigOptions {
        env_prefix: "MYAPP".into(),
        ..Default::default()
    })?;

    // Access config
    let cfg = config::get();
    let db_host = cfg.get_string("database.host").unwrap_or_default();

    // Create metrics
    let metrics_mgr = metrics::MetricsManager::new("myapp");
    let _counter = metrics_mgr.counter("requests_total", "Total requests processed");

    tracing::info!(db_host = %db_host, "Application started");
    Ok(())
}

See docs/CORE-PILLARS.md in the repository for the auto-wiring architecture.

Re-exports

pub use env::Environment;

pub use env::RuntimeContext;

pub use env::runtime_context;

pub use kafka_config::DfeSource;

pub use kafka_config::KafkaConfigError;

pub use kafka_config::KafkaConfigResult;

pub use kafka_config::ServiceRole;

pub use kafka_config::TOPIC_SUFFIX_LAND;

pub use kafka_config::TOPIC_SUFFIX_LOAD;

pub use kafka_config::config_from_file;

pub use kafka_config::config_from_properties_str;

pub use sensitive::SensitiveString;

pub use sensitive::expose_during;

pub use runtime::RuntimePaths;runtime

pub use health::HealthRegistry;health

pub use health::HealthStatus;health

pub use config::Config;config

pub use config::ConfigError;config

pub use config::ConfigOptions;config

pub use config::flat_env::ApplyFlatEnv;config

pub use config::flat_env::EnvVarDoc;config

pub use config::flat_env::Normalize;config

pub use config::reloader::ConfigReloader;config-reload

pub use config::reloader::ReloaderConfig;config-reload

pub use config::shared::SharedConfig;config-reload

pub use logger::LogFormat;logger

pub use logger::LoggerError;logger

pub use logger::LoggerOptions;logger

pub use logger::SecurityEvent;logger

pub use logger::SecurityOutcome;logger

pub use logger::ThrottleConfig;logger

pub use metrics::DfeMetrics;metrics or otel-metrics

pub use metrics::MetricsConfig;metrics or otel-metrics

pub use metrics::MetricsError;metrics or otel-metrics

pub use metrics::MetricsManager;metrics or otel-metrics

pub use metrics::OtelMetricsConfig;otel-metrics

pub use metrics::OtelProtocol;otel-metrics

pub use transport::CommitToken;transport

pub use transport::Message;transport

pub use transport::PayloadFormat;transport

pub use transport::Record;transport

pub use transport::RecordMeta;transport

pub use transport::SendResult;transport

pub use transport::Transport;transport

pub use transport::TransportConfig;transport

pub use transport::TransportError;transport

pub use transport::TransportResult;transport

pub use transport::TransportType;transport

pub use transport::WorkBatch;transport

pub use http_server::HttpServer;http-server

pub use http_server::HttpServerConfig;http-server

pub use http_server::HttpServerError;http-server

pub use spool::Spool;spool

pub use spool::SpoolConfig;spool

pub use spool::SpoolError;spool

pub use tiered_sink::CircuitBreaker;tiered-sink

pub use tiered_sink::CircuitState;tiered-sink

pub use tiered_sink::CompressionCodec;tiered-sink

pub use tiered_sink::DrainStrategy;tiered-sink

pub use tiered_sink::OrderingMode;tiered-sink

pub use tiered_sink::Sink;tiered-sink

pub use tiered_sink::SinkError;tiered-sink

pub use tiered_sink::TieredSink;tiered-sink

pub use tiered_sink::TieredSinkConfig;tiered-sink

pub use tiered_sink::TieredSinkError;tiered-sink

pub use secrets::CacheConfig;secrets

pub use secrets::FileProvider;secrets

pub use secrets::RotationEvent;secrets

pub use secrets::SecretMetadata;secrets

pub use secrets::SecretProvider;secrets

pub use secrets::SecretSource;secrets

pub use secrets::SecretValue;secrets

pub use secrets::SecretsConfig;secrets

pub use secrets::SecretsError;secrets

pub use secrets::SecretsManager;secrets

pub use secrets::SecretsResult;secrets

pub use secrets::OpenBaoAuth;secrets-vault

pub use secrets::OpenBaoConfig;secrets-vault

pub use secrets::OpenBaoProvider;secrets-vault

pub use directory_config::ChangeEvent;directory-config

pub use directory_config::ChangeOperation;directory-config

pub use directory_config::DirectoryConfigError;directory-config

pub use directory_config::DirectoryConfigResult;directory-config

pub use directory_config::DirectoryConfigStore;directory-config

pub use directory_config::DirectoryConfigStoreConfig;directory-config

pub use directory_config::WriteMode;directory-config

pub use directory_config::WriteResult;directory-config

pub use memory::MemoryGuard;memory

pub use memory::MemoryGuardConfig;memory

pub use memory::MemoryPressure;memory

pub use memory::detect_memory_limit;memory

pub use scaling::ComponentSnapshot;scaling

pub use scaling::GateType;scaling

pub use scaling::PressureSnapshot;scaling

pub use scaling::RateWindow;scaling

pub use scaling::ScalingComponent;scaling

pub use scaling::ScalingPressure;scaling

pub use scaling::ScalingPressureConfig;scaling

pub use governor::Admit;governor

pub use governor::ByteBudgetConfig;governor

pub use governor::ByteBudgetController;governor

pub use governor::GateActuator;governor

pub use governor::Hysteresis;governor

pub use governor::InboundGate;governor

pub use governor::MemoryPressureSource;governor

pub use governor::NoopActuator;governor

pub use governor::ObservingActuator;governor

pub use governor::Pressure;governor

pub use governor::PressureSource;governor

pub use governor::SelfRegulationConfig;governor

pub use governor::SelfRegulationGovernor;governor

pub use governor::SelfRegulationProfile;governor

pub use governor::UnifiedPressure;governor

pub use governor::UnifiedPressureSnapshot;governor

pub use worker::AccumulatorConfig;worker-batch or worker-pool or worker

pub use worker::AccumulatorFull;worker-batch or worker-pool or worker

pub use worker::AdaptiveWorkerPool;worker-batch or worker-pool or worker

pub use worker::BatchAccumulator;worker-batch or worker-pool or worker

pub use worker::BatchDrainer;worker-batch or worker-pool or worker

pub use worker::BatchPipeline;worker-batch or worker-pool or worker

pub use worker::BatchProcessor;worker-batch or worker-pool or worker

pub use worker::PipelineStats;worker-batch or worker-pool or worker

pub use worker::PipelineStatsSnapshot;worker-batch or worker-pool or worker

pub use worker::ScalingDecision;worker-batch or worker-pool or worker

pub use worker::ScalingInput;worker-batch or worker-pool or worker

pub use worker::WorkerPoolConfig;worker-batch or worker-pool or worker

pub use cli::CliError;cli

pub use cli::CommonArgs;cli

pub use cli::StandardCommand;cli

pub use cli::VersionInfo;cli

pub use cli::DfeApp;cli-service

pub use cli::ServiceRuntime;cli-service

pub use io::AsyncNdjsonWriter;io

pub use io::FileWriterConfig;io

pub use io::NdjsonWriter;io

pub use io::RotationPeriod;io

pub use dlq::Dlq;dlq

pub use dlq::DlqBackend;dlq

pub use dlq::DlqConfig;dlq

pub use dlq::DlqEntry;dlq

pub use dlq::DlqError;dlq

pub use dlq::DlqMode;dlq

pub use dlq::DlqSource;dlq

pub use dlq::FileDlqConfig;dlq

pub use dlq::HttpDlqConfig;dlq-http

pub use dlq::RedisDlqConfig;dlq-redis

pub use output::FileOutput;output-file

pub use output::FileOutputConfig;output-file

pub use output::OutputError;output-file

pub use expression::ALLOWED_FUNCTIONS;expression

pub use expression::DISALLOWED_FUNCTIONS;expression

pub use expression::ExpressionError;expression

pub use expression::ExpressionResult;expression

pub use expression::build_context;expression

pub use expression::compile;expression

pub use expression::evaluate;expression

pub use expression::evaluate_condition;expression

pub use expression::validate;expression

pub use deployment::ContractMismatch;deployment

pub use deployment::DeploymentContract;deployment

pub use deployment::DeploymentError;deployment

pub use deployment::HealthContract;deployment

pub use deployment::KedaConfig;deployment

pub use deployment::KedaContract;deployment

pub use version_check::VersionCheck;version-check

pub use version_check::VersionCheckConfig;version-check

pub use version_check::VersionCheckResponse;version-check

pub use concurrency::Actor;concurrency

pub use concurrency::ActorConfig;concurrency

pub use concurrency::ActorError;concurrency

pub use concurrency::ActorHandle;concurrency

pub use concurrency::ActorJoinHandle;concurrency

pub use concurrency::BackgroundSink;concurrency

pub use concurrency::BackgroundSinkConfig;concurrency

pub use concurrency::BackgroundSinkHandle;concurrency

pub use concurrency::DrainError;concurrency

pub use concurrency::Overflow;concurrency

pub use concurrency::PeriodicTask;concurrency

pub use concurrency::PeriodicWorker;concurrency

pub use concurrency::SinkDrain;concurrency

pub use concurrency::TickError;concurrency

Modules

cachecache

In-memory cache with per-source TTL, metrics, and invalidation.

clicli

Standard CLI framework for DFE Rust services.

concurrencyconcurrency

Async concurrency primitives.

configconfig

Configuration management with 8-layer cascade.

databasedatabase

Database connection string builders.

deploymentdeployment

Deployment contract validation and generation for Kubernetes/Helm/Docker.

directory_configdirectory-config

Directory Config Store

dlqdlq

Unified dead letter queue (DLQ) with pluggable backends.

env

Runtime environment detection.

expressionexpression

CEL expression evaluation – compile, evaluate, validate.

governorgovernor

Unified self-regulation governor.

healthhealth

Unified health registry for service readiness and liveness.

http_clienthttp

Production HTTP client with automatic retries and timeouts.

http_serverhttp-server

High-performance HTTP server built on axum.

ioio

Shared NDJSON file I/O primitives.

kafka_config

Shared Kafka librdkafka configuration profiles, merge helper, and file loader.

loggerlogger

Structured logging with JSON output and sensitive data masking.

memorymemory

Memory management and OOM prevention.

metricsmetrics or otel-metrics

Metrics with Prometheus and/or OpenTelemetry backends.

otel_tracingotel-tracing

OpenTelemetry distributed tracing – span export via OTLP.

outputoutput-file

File output sink for local NDJSON event writing.

runtimeruntime

Runtime path management.

scalingscaling

Scaling pressure calculation for autoscaler integration.

secretssecrets

Secrets management with multi-provider support and resilient caching.

sensitive

Sensitive string type for fields that must never be exposed.

shutdownshutdown

Unified graceful shutdown manager.

spoolspool

Disk-backed async FIFO queue with optional zstd compression.

strmatchstrmatch

Regex-shaped patterns, fast-path dispatch.

tiered_sinktiered-sink

Tiered sink with automatic disk spillover for resilient message delivery.

transporttransport

Transport Abstraction Layer

version_checkversion-check

Startup version check.

workerworker-batch or worker-pool or worker

Adaptive worker pool and batch processing framework.

Constants

VERSION

Library version

Functions

initconfig and logger

Initialise all library components with default settings.