Crate chie_core

Expand description

Core protocol logic for CHIE Protocol.

This crate provides the main node implementation and content management for the CHIE (Collective Hybrid Intelligence Ecosystem) protocol - a decentralized content distribution system with incentive mechanisms.

§Overview

The chie-core crate contains the fundamental building blocks for running a CHIE node:

Node Management (node): Core node implementation for handling content and proofs
Storage (storage): Chunk-based storage system with encryption support
Protocol (protocol): Bandwidth proof protocol and validation
Content Management (content): Content metadata caching and management
Cryptography (chunk_encryption): Per-chunk encryption utilities
Analytics (analytics): Performance metrics and statistics tracking
Utilities (utils): Helper functions for common operations

§Quick Start

use chie_core::{ContentNode, NodeConfig, PinnedContent};
use std::path::PathBuf;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a node with storage
    let config = NodeConfig {
        storage_path: PathBuf::from("./chie-data"),
        max_storage_bytes: 50 * 1024 * 1024 * 1024, // 50 GB
        max_bandwidth_bps: 100 * 1024 * 1024 / 8,   // 100 Mbps
        coordinator_url: "https://coordinator.chie.network".to_string(),
    };

    let mut node = ContentNode::with_storage(config).await?;

    // Pin content for distribution
    let content = PinnedContent {
        cid: "QmExample123".to_string(),
        size_bytes: 1024 * 1024,
        encryption_key: [0u8; 32],
        predicted_revenue_per_gb: 10.0,
    };
    node.pin_content(content);

    println!("Node public key: {:?}", node.public_key());
    println!("Pinned content count: {}", node.pinned_count());

    Ok(())
}

§Features

§Content Storage

The storage system splits content into chunks, encrypts each chunk individually, and stores them with cryptographic verification:

use chie_core::{ChunkStorage, split_into_chunks};
use chie_crypto::{generate_key, generate_nonce};
use std::path::PathBuf;

let storage_path = PathBuf::from("./storage");
let max_bytes = 10 * 1024 * 1024 * 1024; // 10 GB

let mut storage = ChunkStorage::new(storage_path, max_bytes).await?;

// Split and store content
let data = b"Hello, CHIE Protocol!";
let chunks = split_into_chunks(data, 1024);
let key = generate_key();
let nonce = generate_nonce();

storage.pin_content("QmTest", &chunks, &key, &nonce).await?;

§Bandwidth Proof Protocol

Nodes earn rewards by serving content and generating cryptographically-signed bandwidth proofs:

use chie_core::{create_chunk_request, create_bandwidth_proof};
use chie_crypto::KeyPair;

let requester_keypair = KeyPair::generate();
let provider_keypair = KeyPair::generate();

// Create a chunk request with challenge nonce
let request = create_chunk_request(
    "QmContent123".to_string(),
    0, // chunk index
    "requester-peer-id".to_string(),
    requester_keypair.public_key(),
);

// Provider serves chunk and signs response
// Requester verifies and co-signs
// Both submit dual-signed proof to coordinator

§Performance Optimization

The crate includes several performance optimization features:

Chunk Prefetching (prefetch): Predict and preload chunks based on access patterns
Tiered Storage (tiered_storage): Hot/warm/cold storage tiers for cost optimization
Rate Limiting (ratelimit): Per-peer bandwidth throttling
Content Deduplication (dedup): Avoid storing duplicate chunks
Garbage Collection (gc): Remove unprofitable content automatically

§Module Overview

adaptive_ratelimit - Adaptive rate limiting with dynamic adjustments
analytics - Performance metrics and earnings tracking
anomaly - Anomaly detection for fraud prevention
auto_repair - Automatic data integrity repair for corrupted chunks
backup - Storage backup and recovery utilities
batch - Batch processing utilities
cache - Advanced caching with TTL and memory management
cache_invalidation - Distributed cache invalidation notifications
chunk_encryption - Encrypted chunk data structures
circuit_breaker - Circuit breaker pattern for resilient service calls
compression - Content compression utilities
connection_multiplexing - HTTP connection pooling and multiplexing for coordinator communication
content - Content metadata management
content_aware_cache - Content-aware cache sizing with intelligent management
content_router - Content routing optimizer
custom_exporters - Custom metrics exporters (StatsD, InfluxDB)
dedup - Content deduplication
gc - Garbage collection for storage
health - Health check system for node monitoring
http_pool - HTTP connection pooling utilities
integrity - Content integrity verification
lifecycle - Content lifecycle events and webhooks
metrics - Prometheus-compatible metrics exporter
network_diag - Network diagnostics and monitoring
node - Core node implementation
peer_selection - Intelligent peer selection and ranking
pinning - Selective pinning optimizer
popularity - Content popularity tracking
prefetch - Chunk prefetching
profiler - Performance profiling and timing utilities
proof_submit - Proof submission with retry logic
protocol - Bandwidth proof protocol
qos - Quality of Service with priority-based scheduling
quic_transport - QUIC transport integration for modern, efficient networking
ratelimit - Rate limiting
reputation - Peer reputation tracking
resource_mgmt - Advanced resource management and monitoring
storage - Chunk storage backend
storage_health - Storage health monitoring with predictive failure detection
streaming - Streaming utilities for large content
tiered_storage - Multi-tier storage system
tracing - OpenTelemetry tracing integration for distributed observability
utils - Utility functions
validation - Content and proof validation utilities

Re-exports§

pub use adaptive_ratelimit::*;
pub use adaptive_retry::*;
pub use alerting::*;
pub use analytics::*;
pub use anomaly::*;
pub use auto_repair::*;
pub use backup::*;
pub use batch::*;
pub use cache::*;
pub use cache_admission::*;
pub use cache_invalidation::*;
pub use cache_warming::*;
pub use checkpoint::*;
pub use chunk_encryption::*;
pub use circuit_breaker::*;
pub use compression::*;
pub use config::*;
pub use connection_multiplexing::*;
pub use content::*;
pub use content_aware_cache::*;
pub use content_router::*;
pub use dashboard::*;
pub use dedup::*;
pub use degradation::*;
pub use events::*;
pub use expiration::*;
pub use forecasting::*;
pub use gc::*;
pub use geo_selection::*;
pub use health::*;
pub use http_pool::*;
pub use integrity::*;
pub use lifecycle::*;
pub use logging::*;
pub use metrics::*;
pub use metrics_exporter::*;
pub use network_diag::*;
pub use node::*;
pub use orchestrator::*;
pub use partial_chunk::*;
pub use peer_selection::*;
pub use pinning::*;
pub use popularity::*;
pub use prefetch::*;
pub use priority_eviction::*;
pub use profiler::*;
pub use proof_submit::*;
pub use protocol::*;
pub use qos::*;
pub use ratelimit::*;
pub use reputation::*;
pub use request_pipeline::*;
pub use resource_mgmt::*;
pub use storage::*;
pub use storage_health::*;
pub use streaming::*;
pub use streaming_verification::*;
pub use test_utils::*;
pub use tier_migration::*;
pub use tiered_storage::*;
pub use transaction::*;
pub use utils::*;
pub use validation::*;
pub use wal::*;

Modules§

adaptive_ratelimit: Adaptive rate limiting with dynamic adjustments based on peer behavior.
adaptive_retry: Adaptive retry policies based on failure patterns.
alerting: Alerting system with configurable thresholds.
analytics: Local analytics for CHIE node dashboard.
anomaly: Anomaly detection module for identifying suspicious behavior and fraud.
auto_repair: Automatic data integrity repair for corrupted chunks.
backup: Backup and restore functionality for CHIE storage.
bandwidth_estimation: Bandwidth estimation and congestion detection for the CHIE protocol.
batch: Batch processing utilities for parallel operations.
cache: Advanced caching utilities with TTL and memory management.
cache_admission: Cache admission policies for intelligent caching decisions.
cache_invalidation: Distributed cache invalidation notifications.
cache_warming: Cache warming strategies for cold starts.
checkpoint: Checkpoint-based state recovery system.
chunk_encryption: Chunk-level encryption with per-chunk nonces for CHIE Protocol.
circuit_breaker: Circuit breaker pattern for resilient external service calls.
compression: Content compression utilities for storage optimization.
config: Configuration management for CHIE node settings.
connection_multiplexing: Connection multiplexing for coordinator communication.
content: Content management for CHIE Protocol.
content_aware_cache: Content-aware cache sizing with intelligent memory management.
content_router: Content routing optimizer for efficient content discovery and retrieval.
custom_exporters: Custom metrics exporters for StatsD and InfluxDB.
dashboard: Performance dashboard data endpoints.
dedup: Content deduplication for storage efficiency.
degradation: Graceful degradation under resource pressure.
events: Event bus for internal pub/sub communication.
expiration: Automatic content expiration policies.
forecasting: Resource usage forecasting with time series analysis.
gc: Garbage collection for unprofitable content.
geo_selection: Geographic-aware peer selection for optimal content delivery.
health: Health check system for monitoring node components.
http_pool: HTTP connection pooling utilities.
integrity: Content integrity verification for CHIE Protocol.
lifecycle: Content lifecycle event system for webhooks and callbacks.
logging: Logging configuration with configurable verbosity.
metrics: Prometheus-compatible metrics exporter for observability.
metrics_exporter: Custom metrics exporters for external monitoring systems.
network_diag: Network diagnostics and monitoring utilities.
node: Node management for CHIE Protocol.
orchestrator: Request orchestration for intelligent content retrieval.
partial_chunk: Partial chunk support for range requests.
peer_selection: Intelligent peer selection module for optimizing content delivery.
pinning: Selective pinning optimizer for CHIE Protocol.
popularity: Content popularity tracking for CHIE Protocol.
prefetch: Chunk prefetching for improved performance.
priority_eviction: Priority-based content eviction.
profiler: Performance profiling utilities for CHIE Protocol.
proof_submit: Proof submission with retry logic.
protocol: Bandwidth proof protocol implementation.
qos: Quality of Service (QoS) for priority-based request handling.
quic_transport: QUIC Transport Integration
ratelimit: Bandwidth rate limiting for P2P transfers.
reputation: Peer reputation tracking system for network reliability.
request_pipeline: Request pipelining for efficient batch operations.
resource_mgmt: Advanced resource management and monitoring.
storage: Chunk storage and retrieval for CHIE Protocol.
storage_health: Storage health monitoring with predictive failure detection.
streaming: Streaming utilities for large content transfers.
streaming_verification: Incremental content verification with streaming hash.
system_coordinator: System-wide coordinator for integrated monitoring and management.
test_utils: Test utilities and helpers for testing.
tier_migration: Content migration between storage tiers.
tiered_cache: Multi-level cache hierarchy for optimal performance and cost.
tiered_storage: Tiered storage for CHIE nodes.
tracing: OpenTelemetry tracing integration for distributed observability.
transaction: Transactional chunk operations for atomic storage writes.
utils: Utility functions for CHIE Protocol operations.
validation: Content validation utilities for CHIE Protocol.
wal: Write-Ahead Logging (WAL) for crash recovery.

Macros§

log_debug: Macro for logging debug with automatic module path.
log_error: Macro for logging with automatic module path.
log_info: Macro for logging info with automatic module path.
log_trace: Macro for logging trace with automatic module path.
log_warn: Macro for logging warnings with automatic module path.
profile: Macro for easy profiling of code blocks.

Crate chie_core

Crate chie_core Copy item path

§Overview

§Quick Start

§Features

§Content Storage

§Bandwidth Proof Protocol

§Performance Optimization

§Module Overview

Re-exports§

Modules§

Macros§

Crate chie_core