# Rialo Aggregator Utils
This crate provides essential utilities for building robust aggregator programs in the Rialo ecosystem. It offers
statistical functions, data processing tools, and error handling utilities to simplify the implementation of aggregator
programs that process oracle data.
## Overview
Rialo Aggregator Utils serves as a foundation library for developers building aggregator programs. It focuses on
providing robust statistical methods and data extraction utilities that handle real-world challenges like outliers,
malformed data, and varying data formats from multiple oracle sources.
> Note the current implementation is focused on median calculation, with plans to extend to other statistical methods
> in the future. For future extensions ideas, please refer to the [Future Extensions](#future-extensions) section.
## Core Features
### Median Calculation
The primary utility is a sophisticated median calculation system designed for processing oracle price data:
- **Multiple calculation methods** for handling even-sized datasets
- **Robust error handling** with detailed error reporting
- **NaN and outlier resistance** for reliable results
- **Flexible data extraction** from Base64-encoded JSON oracle updates
### Error Handling
Comprehensive error handling for common oracle data issues:
- **Base64 decoding errors**
- **JSON parsing failures**
- **Missing data fields**
- **Non-numeric values**
- **Detailed error tracking** with index mapping
## Median Calculation
### MedianMethod Enum
Defines how to calculate the median when dealing with an even number of values:
```rust
pub enum MedianMethod {
/// Average the two middle values: (a + b) / 2 (default)
Average,
/// Take the lower of the two middle values
Lower,
/// Take the upper of the two middle values
Upper,
}
```
### Median Struct
The core result structure that implements the `RialoEvent` trait:
```rust
#[derive(Clone, Debug, Default, PartialEq, RialoEvent)]
pub struct Median {
pub median: f64,
pub sample_size: usize,
}
```
### MedianResult
Comprehensive result type that includes both the calculated median and any errors encountered:
```rust
pub struct MedianResult {
pub median: Median,
pub errors: Vec<(usize, ValueExtractionError)>,
}
```
## Usage Examples
### Basic Median Calculation
```rust
use rialo_aggregator_utils::median::{Median, MedianMethod};
use rialo_oracle_processor_interface::state::OracleUpdate;
// Calculate median with default method (Average)
let result = Median::from_with_method("price", &oracle_updates, MedianMethod::Average);
// Access the median value
let median = result.median;
println!("Median price: {}, Sample size: {}", median.median, median.sample_size);
// Check for errors during processing
for (index, error) in result.errors() {
println!("Error at index {}: {}", index, error);
}
```
### Different Median Methods
```rust
// For even-sized datasets, different methods yield different results
let updates = vec![/* oracle updates with values [10, 20, 30, 40] */];
// Average method: (20 + 30) / 2 = 25.0
let avg_result = Median::from_with_method("price", &updates, MedianMethod::Average);
// Lower method: 20.0
let lower_result = Median::from_with_method("price", &updates, MedianMethod::Lower);
// Upper method: 30.0
let upper_result = Median::from_with_method("price", &updates, MedianMethod::Upper);
```
### Error-Resilient Processing
```rust
// Process oracle updates with potential errors
let result = Median::from_with_method("price", &oracle_updates, MedianMethod::Average);
// The median is calculated from valid values only
println!("Calculated median: {}", result.median.median);
// Inspect any errors that occurred
match result.errors().len() {
0 => println!("All oracle updates processed successfully"),
n => {
println!("Encountered {} errors during processing:", n);
for (index, error) in result.errors() {
match error {
ValueExtractionError::DecodeError(_) => {
println!(" Oracle {} had invalid Base64 data", index);
}
ValueExtractionError::JsonError(_) => {
println!(" Oracle {} had malformed JSON", index);
}
ValueExtractionError::MissingKey(key) => {
println!(" Oracle {} missing key: {}", index, key);
}
ValueExtractionError::NotANumber(val) => {
println!(" Oracle {} had non-numeric value: {}", index, val);
}
}
}
}
}
```
### Backward Compatibility
The crate provides convenient `From` trait implementations for simple usage:
```rust
// Direct conversion for simple cases
let median: Median = ("price", &oracle_updates).into();
// Or get the full result with error information
let result: MedianResult = ("price", &oracle_updates).into();
```
## Data Processing Pipeline
The median calculation follows this robust pipeline:
1. **Data Extraction**: Decode Base64 oracle data and parse JSON
2. **Value Extraction**: Extract numeric values for the specified key
3. **Error Collection**: Track failed extractions with detailed error information
4. **Statistical Calculation**: Compute median using the specified method
5. **Result Packaging**: Return both the median and error details
### Oracle Data Format
The utility expects oracle updates with Base64-encoded JSON data:
```json
{
"price": 50000.0,
"timestamp": 1634567890,
"source": "exchange_a"
}
```
The `extract_value` function handles:
- **Base64 decoding** of the oracle data
- **JSON parsing** with error recovery
- **Type conversion** from JSON numbers to f64
- **Missing key detection** and reporting
## Error Types
### ValueExtractionError
Comprehensive error types for different failure modes:
```rust
pub enum ValueExtractionError {
/// Base64 decoding failed
DecodeError(DecodeError),
/// JSON parsing failed
JsonError(serde_json::Error),
/// Required key missing from JSON
MissingKey(String),
/// Value exists but is not numeric
NotANumber(String),
}
```
Each error provides detailed context for debugging and monitoring oracle data quality.
## Robustness Features
### NaN Handling
- **Automatic filtering** of NaN values from calculations
- **Safe comparison** functions that handle NaN edge cases
- **Error reporting** for NaN values in source data
### Outlier Resistance
- **Median-based approach** naturally resistant to extreme outliers
- **Configurable methods** for different statistical requirements
- **Sample size tracking** for confidence assessment
### Data Quality Monitoring
- **Error indexing** to identify problematic oracle sources
- **Detailed error types** for targeted debugging
- **Partial result computation** when some oracles fail
## Integration with Rialo Ecosystem
### Event System Integration
The `Median` struct implements `RialoEvent`, enabling direct emission:
```rust
// Calculate median and emit as event
let result = Median::from_with_method("price", &updates, MedianMethod::Average);
result.median.emit(program_id, &accounts)?;
```
### Aggregator Program Usage
Perfect for use in aggregator programs:
```rust
// In your aggregator program
let oracle_report_data = deserialize_oracle_report(&oracle_account)?;
let median_result = Median::from_with_method(
"price",
&oracle_report_data.into_updates(),
MedianMethod::Average
);
// Use the median in your business logic
let trading_pairs = TradingPairs::from_median_results(median_result);
```
## Future Extensions
This crate is designed to be extensible with additional utilities:
- **Other robust statistics** (trimmed mean, weighted median, etc.)
- **Time-series analysis** utilities
- **Data validation** and quality scoring
- **Multi-key aggregation** helpers
- **Custom aggregation strategies**
## Testing
The crate includes comprehensive tests covering:
- **Different median methods** with various data sizes
- **Error handling scenarios** for all error types
- **Edge cases** (empty data, single values, NaN handling)
- **Performance testing** with large datasets
- **Data format variations** (integers, floats, mixed types)
## Dependencies
- `base64`: Base64 encoding/decoding for oracle data
- `rialo_events_core`: Core event framework and derive macros
- `rialo_oracle_processor_interface`: Oracle update data structures
- `serde_json`: JSON parsing and manipulation
- `thiserror`: Error handling and derivation
## License
Copyright (c) Subzero Labs, Inc.
SPDX-License-Identifier: Apache-2.0