ETL Core

Core ETL (Extract, Transform, Load) library for Rust.

Overview

This library provides a framework for building ETL pipelines in Rust. It includes traits and implementations for extracting data from various sources, transforming it, and loading it into various destinations.

Features

Extensible extractor framework with built-in REST API support
Error handling with detailed error types
Asynchronous operations using Tokio
Support for various data formats (JSON, text, bytes)

Modules

extract: Traits and implementations for data extraction

Usage

Add this to your Cargo.toml:

[dependencies]
etl-core = "0.1"

Or if you want to use the latest version from the repository:

[dependencies]
etl-core = { git = "https://github.com/your-username/anduflow", branch = "main" }

Examples

use etl_core::extract::{Extractor, ExtractorResult, rest_extractor::RestExtractor};

#[tokio::main]
async fn main() -> ExtractorResult<()> {
    let extractor = RestExtractor::new("https://api.example.com", "data");
    let data: serde_json::Value = extractor.extract().await?;
    Ok(())
}

Testing

To run the tests:

cargo test

Tests are organized in a separate tests directory for better maintainability:

Unit tests for error handling in tests/error_tests.rs
Integration tests for REST extractor in tests/rest_extractor_tests.rs

Documentation

To generate and view the documentation:

cargo doc --open

For online documentation, visit docs.rs/etl-core.

etl-core 0.1.0