dmsc 0.1.8

Dunimd Middleware Service - A high-performance Rust middleware framework with modular architecture
Documentation

English | 简体中文

Help Documentation | Changelog | Security | Contributing | Code of Conduct

DMSC (Dunimd Middleware Service) — A high-performance Rust middleware framework that unifies backend infrastructure. Built for enterprise-scale with modular architecture, built-in observability, and distributed systems support.

📐 Modular Design

DMSC adopts a highly modular architecture with 18 core modules, enabling on-demand composition and seamless extension:

Module Description
auth Authentication & authorization (JWT, OAuth, permissions)
cache Multi-backend cache abstraction (Memory, Redis, Hybrid)
config Multi-source configuration management with hot reload
core Runtime, error handling, and service context
database Database abstraction with PostgreSQL, MySQL, SQLite support
device Device control, discovery, and intelligent scheduling
fs Secure file system operations and management
gateway API gateway with load balancing, rate limiting, and circuit breaking
grpc gRPC server and client support with Python bindings (requires grpc feature)
hooks Lifecycle event hooks (Startup, Shutdown, etc.)
log Structured logging with tracing context integration
module_rpc Inter-module RPC communication for distributed method calls
observability Metrics, tracing, and Grafana integration
database.orm Type-safe ORM with repository pattern, query builder, and Python bindings
protocol Protocol abstraction layer for multi-protocol support (requires pyo3 feature)
queue Distributed queue abstraction (Kafka, RabbitMQ, Redis, Memory)
service_mesh Service discovery, health checking, and traffic management
validation Input validation and data sanitization utilities
ws WebSocket server support with Python bindings (requires websocket feature)
c C/C++ FFI bindings for cross-language integration (requires c feature)
java Java JNI bindings for Java application integration (requires java feature)

Note: Some modules require specific feature flags:

  • grpc: gRPC support (--features grpc)
  • websocket: WebSocket support (--features websocket)
  • protocol: Protocol abstraction layer (--features protocol or full)
  • c: C/C++ FFI bindings (--features c)
  • java: Java JNI bindings (--features java)

🚀 Key Features

🔍 Distributed Tracing

  • W3C Trace Context standard implementation
  • Full-chain TraceID/SpanID propagation
  • Baggage data transmission for business context
  • Multi-language compatibility (Java, Go, Python)
  • Automatic span creation via #[tracing::instrument] attribute

📊 Enterprise Observability

  • Native Prometheus metrics export
  • Counter, Gauge, Histogram, Summary metric types
  • Out-of-the-box Grafana dashboard integration
  • Real-time performance statistics with quantile calculation
  • Full-stack metrics (CPU, memory, I/O, network)

🤖 Intelligent Device Management

  • Auto-discovery and registration
  • Efficient resource pool management
  • Policy-based scheduling with priority support
  • Dynamic load balancing
  • Complete device lifecycle management

📝 Structured Logging

  • JSON and text format support
  • Configurable sampling rates
  • Intelligent log rotation
  • Automatic tracing context inclusion
  • DEBUG/INFO/WARN/ERROR log levels

⚙️ Flexible Configuration

  • Multi-source loading (files, environment variables, runtime)
  • Hot configuration updates
  • Modular architecture for on-demand composition
  • Plugin-based extension mechanism

📁 Secure File System

  • Unified project root directory management
  • Atomic file operations
  • Categorized directory structure
  • JSON data persistence
  • Secure path handling

Prerequisites

  • Rust: 1.65+ (2021 Edition)
  • Cargo: 1.65+
  • Platforms: Linux (x64, arm64), macOS (x64, arm64), Windows (x64, arm64), Android (arm64-v8a, armeabi-v7a, x86_64)

Build Dependencies

Some features require additional system dependencies:

Dependency Required For Installation
protoc etcd feature (Protocol Buffers) Protocol Buffers
CMake + C++ compiler kafka feature (rdkafka) See instructions below

Installing protoc

Windows:

# Using chocolatey
choco install protoc

# Or download from GitHub releases
# https://github.com/protocolbuffers/protobuf/releases

macOS:

brew install protobuf

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install -y protobuf-compiler

Linux (CentOS/RHEL):

sudo yum install -y protobuf-compiler

Installing CMake and C++ compiler (for Kafka support)

Windows:

# CMake is usually installed with Visual Studio
# Or download from: https://cmake.org/download/

# Using chocolatey
choco install cmake

macOS:

# CMake and C++ compiler (Xcode Command Line Tools)
xcode-select --install

# Or using Homebrew
brew install cmake

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install -y cmake build-essential

Linux (CentOS/RHEL):

sudo yum install -y cmake gcc-c++ make

Quick Setup

Add DMSC to your project's Cargo.toml:

[dependencies]
dmsc = "0.1.8"

Or use cargo add:

cargo add dmsc

Core API Usage

use dmsc::prelude::*;

#[tokio::main]
async fn main() -> DMSCResult<()> {
    // Build service runtime
    let app = DMSCAppBuilder::new()
        .with_config("config.yaml")?
        .with_logging(DMSCLogConfig::default())
        .with_observability(DMSCObservabilityConfig::default())
        .build()?;
    
    // Run business logic
    app.run(|ctx: &DMSCServiceContext| async move {
        ctx.logger().info("service", "DMSC service started")?;
        // Your business code here
        Ok(())
    }).await
}

Observability Example

use dmsc::prelude::*;
use dmsc::observability::{DMSCTracer, DMSCSpanKind, DMSCSpanStatus};

#[tracing::instrument(name = "user_service", skip(ctx))]
async fn get_user(ctx: &DMSCServiceContext, user_id: u64) -> DMSCResult<User> {
    let user = fetch_user_from_db(user_id).await?;
    Ok(user)
}

Or using DMSCTracer directly:

use dmsc::prelude::*;
use dmsc::observability::DMSCTracer;

async fn get_user(ctx: &DMSCServiceContext, user_id: u64) -> DMSCResult<User> {
    let tracer = DMSCTracer::new(1.0);
    let _span = tracer.span("get_user")
        .with_attribute("user_id", user_id.to_string())
        .start();
    let user = fetch_user_from_db(user_id).await?;
    Ok(user)
}

Configuration Example

# config.yaml
service:
  name: "my-service"
  version: "1.0.0"

logging:
  level: "info"
  file_format: "json"
  file_enabled: true
  console_enabled: true

observability:
  metrics_enabled: true
  tracing_enabled: true
  prometheus_port: 9090

resource:
  providers: ["cpu", "gpu", "memory"]
  scheduling_policy: "priority_based"

Configuration Sources

DMSC supports multiple configuration sources in order of priority (lowest to highest):

  1. Configuration files (YAML, TOML, JSON)
  2. Custom configuration via code
  3. Environment variables (prefixed with DMSC_)

Running Tests

Multi-Language Testing

DMSC provides comprehensive testing across all supported languages:

  • Rust: Core library tests with cargo test
  • Python: Python binding tests with pytest
  • Java: JNI binding tests with standard Java test runner
  • C/C++: C API tests with native compilers

Test Coverage

The tests verify:

  • ✅ Builder pattern behavior in all languages
  • ✅ Method chaining returns appropriate instances (language-specific)
  • ✅ Runtime creation and lifecycle management
  • ✅ Error handling and edge cases
  • ✅ Cross-language API consistency

Running Rust Tests

# Run all Rust tests
cargo test

# Run specific test modules
cargo test --lib app_builder
cargo test --lib app_runtime

# Run with verbose output
cargo test -- --nocapture

# Run with all features
cargo test --all-features

Running Python Tests

# Install Python package in development mode
cd python
pip install -e .

# Run all Python tests
python -m pytest tests/Python/ -v

# Run specific test classes
python -m pytest tests/Python/test_core.py::TestDMSCAppBuilderWrapper -v
python -m pytest tests/Python/test_core.py::TestDMSCAppRuntimeWrapper -v

Running Java Tests

# Build JNI library
cargo build --release --no-default-features --features java

# Compile and run Java tests
cd java
javac -d build/classes/java/test -cp build/classes/java/main \
  src/test/java/TestAll.java src/test/java/TestAppBuilder.java

java -cp build/classes/java/test:build/classes/java/main \
  -Djava.library.path=../target/release TestAll

API Behavior Across Languages

Language Builder Pattern Method Chaining Reason
Rust Returns Self Consumes original Native builder pattern
Python Returns self Same instance Pythonic wrapper for PyO3
Java Returns new instance Immutable builder Java best practice
C Returns new pointer Memory management C idioms

Q: How to add a new module? A: Implement the DMSCModule trait and register it via DMSCAppBuilder::with_module.

Q: How to configure logging level? A: Set logging.level in the configuration file, supporting DEBUG/INFO/WARN/ERROR levels.

Q: How to enable metrics export? A: Set observability.metrics_enabled: true and configure prometheus_port in the configuration file.

Q: How to extend configuration sources? A: Implement a custom configuration loader and register it with DMSCConfigManager.

Q: How to handle asynchronous tasks? A: Use DMSCAppBuilder::with_async_module to add async modules, the framework handles async lifecycle automatically.

📄 License & Open Source Agreements

🏛️ Project License

This project uses Apache License 2.0 open source agreement, see LICENSE file.

📋 Dependency Package Open Source Agreements

Open source packages and their agreement information used by this project:

Dependencies License

📦 Package 📜 License 📦 Package 📜 License
serde MIT/Apache-2.0 serde_json MIT/Apache-2.0
serde_yaml MIT/Apache-2.0 tokio MIT
futures MIT/Apache-2.0 futures-util MIT/Apache-2.0
http MIT/Apache-2.0 hyper MIT
prometheus MIT/Apache-2.0 redis MIT
lapin MIT/Apache-2.0 rdkafka MIT
yaml-rust MIT/Apache-2.0 toml MIT/Apache-2.0
etcd-client MIT sysinfo MIT
async-trait MIT/Apache-2.0 dashmap MIT
chrono MIT uuid MIT/Apache-2.0
rand MIT/Apache-2.0 notify CC0-1.0
jsonwebtoken MIT reqwest MIT/Apache-2.0
urlencoding MIT parking_lot MIT/Apache-2.0
log MIT/Apache-2.0 tracing MIT
pyo3 MIT/Apache-2.0 jni MIT/Apache-2.0
safer-ffi MIT tempfile MIT/Apache-2.0
thiserror MIT/Apache-2.0 hex MIT/Apache-2.0
base64 MIT/Apache-2.0 regex MIT/Apache-2.0
url MIT/Apache-2.0 aes-gcm MIT/Apache-2.0
ring ISC lazy_static MIT/Apache-2.0
libloading ISC zeroize MIT/Apache-2.0
zeroize_derive MIT/Apache-2.0 secrecy MIT
erased-serde MIT data-encoding MIT
crc32fast MIT/Apache-2.0 generic-array MIT
bincode MIT typenum MIT/Apache-2.0
html-escape MIT rustls MIT/Apache-2.0
rustls-pemfile MIT/Apache-2.0 webpki ISC
rustls-native-certs MIT/Apache-2.0 tokio-rustls MIT/Apache-2.0
bytes MIT tonic MIT
prost MIT/Apache-2.0 prost-types MIT/Apache-2.0
tokio-stream MIT tower MIT
async-stream MIT tokio-tungstenite MIT
tungstenite MIT num-bigint MIT/Apache-2.0
oqs MIT/Apache-2.0 sm-crypto MIT
openssl-sys Apache-2.0 tokio-postgres MIT/Apache-2.0
rusqlite MIT sqlx MIT/Apache-2.0
criterion MIT/Apache-2.0