Expand description
Data pipeline APIs
Provides a flexible framework for building data processing pipelines:
- Composable pipeline stages for reading, transforming, and writing data
- Multiple execution strategies (sequential, parallel, streaming, async)
- Built-in transformations (normalization, encoding, aggregation)
- Error handling and recovery mechanisms
- Progress tracking and monitoring
- Caching and checkpointing for long-running pipelines Data pipeline APIs for building complex data processing workflows
Provides a flexible framework for constructing data processing pipelines with:
- Composable pipeline stages
- Error handling and recovery
- Progress tracking and monitoring
- Parallel and streaming execution
- Caching and checkpointing
Modules§
- config
- Configuration types and structures for advanced pipeline optimization
- monitoring
- System resource monitoring for pipeline optimization
- neuromorphic
- Neuromorphic computing engine for bio-inspired optimization
- performance
- Performance tracking and auto-tuning for pipeline optimization
- quantum
- Quantum-inspired optimization algorithms for pipeline optimization
Structs§
- Adaptation
Event - Aggregate
Transform - Aggregation transformer
- Aggregation
Stage - Aggregation stage
- Alert
- Auto
Tuner - Machine learning-based auto-tuner for optimal parameter selection
- Auto
Tuning Parameters - Auto-tuning parameters for optimization
- Backpressure
Streaming Executor - Enhanced streaming executor with backpressure control
- Branching
Pipeline Builder - Builder for branching pipelines
- Cache
Configuration - Cache configuration for optimal data locality
- Cache
Performance - Cache
Stage - Cache stage - caches intermediate results
- Cached
Executor - Cached executor - caches intermediate results
- Checkpointed
Executor - Checkpointed executor - saves progress at intervals
- Consciousness
Config - Consciousness-inspired optimization configuration
- Data
Lineage - Data lineage tracker for tracking data transformations
- Distributed
Executor - Distributed executor - distributes work across multiple workers
- Encoding
Transform - Encoding transformer for categorical data
- Enrichment
Stage - Enrichment stage - adds metadata or augments data
- Error
Handling Stage - Error handling stage - catches and handles errors
- Event
Driven Executor - Event-driven executor that triggers on specific conditions
- Execution
Record - Individual execution record for performance tracking
- Executor
Factory - Factory for creating executors
- Fallback
Error Handler - Fallback error handler that provides default values
- Feature
Engineering Transform - Feature engineering transformer
- File
Read Stage - File reading stage
- File
Write Stage - File writing stage
- Filter
Stage - Filtering stage
- Impute
Transform - Missing value imputation transformer
- Memory
Usage - Monitoring
Executor - Monitoring executor that collects detailed metrics
- Monitoring
Stage - Monitoring stage - logs metrics and progress
- Monitoring
Thresholds - Neuromorphic
Config - Neuromorphic optimization configuration
- Neuromorphic
Memory - Neuromorphic memory for experience retention
- Neuromorphic
Optimizer - Neuromorphic Computing Engine for Bio-Inspired Optimization
- Normalize
Transform - Normalization transformer
- Numa
Node - Numa
Topology - Optimization
Report - Optimization
Suggestion - Optimized
Pipeline Config - Optimized pipeline configuration with advanced settings
- Outlier
Transform - Outlier detection and removal transformer
- PCATransform
- Principal Component Analysis transformer
- Parallel
Pipeline Builder - Builder for parallel pipelines
- Parallel
Stage Executor - Parallel stage executor for executing pipeline stages in parallel
- Performance
History - Performance history tracker for machine learning optimization
- Performance
Predictor - Performance predictor for estimating pipeline execution time and resource usage
- Performance
Trend - Pipeline
- Main pipeline structure
- Pipeline
Builder - Fluent builder for constructing pipelines
- Pipeline
Composer - Pipeline composition for combining multiple pipelines
- Pipeline
Config - Pipeline configuration
- Pipeline
Context - Pipeline execution context
- Pipeline
Data - Pipeline data wrapper that carries data and metadata through stages
- Pipeline
Metrics - Pipeline
Monitor - Pipeline monitoring and alerting
- Pipeline
Optimizer - Pipeline optimizer for reordering stages
- Pipeline
Performance Metrics - Pipeline performance metrics for optimization feedback
- Pipeline
Profile - Pipeline performance profile with statistical analysis
- Pipeline
Stats - Pipeline execution statistics
- Quantum
Annealer - Quantum annealing simulator for global optimization
- Quantum
Optimization Config - Quantum optimization configuration
- Quantum
Optimizer - Quantum optimization engine for pipeline parameters
- Quantum
State - Quantum state representation for optimization problems
- Regression
Detector - Performance regression detector using statistical methods
- Reshape
Transform - Reshape transformer
- Resource
Monitor - Real-time system resource monitor
- Retry
Error Handler - Default error handler that logs and retries
- Retry
Executor - Retry executor for fault tolerance
- Sequential
Executor - Sequential executor - executes stages one after another
- Serialized
Pipeline - Pipeline serialization for saving/loading pipeline configurations
- Serialized
Stage - Skip
Error Handler - Skip error handler that continues on error
- Spike
Pattern - Spiking
Neural Network - Spiking Neural Network for temporal processing
- Spiking
Neuron - Individual spiking neuron with temporal dynamics
- Stage
Metrics - Streaming
Executor - Streaming executor - processes data in chunks
- Synaptic
Connection - Synaptic connection between neurons
- System
Metrics - System resource metrics
- Text
Processing Transform - Text processing transformer
- Transform
Stage - Data transformation stage
- Transformation
Record - Type
Convert Transform - Type conversion transformer
- Validation
Stage - Data validation stage
Enums§
- Aggregation
Method - Alert
Severity - Batch
Processing Mode - Batch processing mode configuration
- Binning
Strategy - Cache
Replacement Policy - Encoding
Method - Event
- Feature
Operation - File
Format - Imputation
Strategy - Memory
Strategy - Memory allocation and management strategy
- Normalization
Method - Outlier
Method - Pipeline
Error - Pipeline-specific error type
- Plasticity
Rule - Plasticity rules for synaptic adaptation
- Prefetch
Strategy - Data prefetch strategy for reducing memory latency
- Text
Operation - Trend
Direction
Traits§
- Data
Enricher - Data
Transformer - Error
Handler - Monitor
- Pipeline
Executor - Trait for pipeline executors
- Pipeline
Stage - Trait for pipeline stages
- Validator
Functions§
- function_
stage - Create a simple function-based pipeline stage
Type Aliases§
- Pipeline
Result - Result type for pipeline operations