flyby 0.1.1

A high-performance Rust framework for composable data-ingestion pipelines.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94

#![forbid(unsafe_code)]
#![doc = include_str!("../../docs/core/README.md")]
//!
//! # flyby::core
//!
//! Platform-independent core of the FlyBy framework.
//!
//! This module defines the stable programming model that every backend
//! (AF_XDP, io_uring, DPDK, SPDK, simulator, future transports) must
//! implement. It deliberately contains **no** backend-specific code and
//! **no** `unsafe` blocks: it is the contract layer, not the
//! implementation layer.
//!
//! The guiding abstraction is:
//!
//! ```text
//! Source -> Decode -> Transform -> Route -> Sink
//! ```
//!
//! Backends may change. The traits in this crate should not, without an
//! accompanying Architecture Decision Record (ADR).
//!
//! ## Contents
//!
//! | Module      | Purpose                                                  |
//! |-------------|----------------------------------------------------------|
//! | [`message`](crate::core::message) | The [`Message`] trait: typed records flowing downstream. |
//! | [`source`](crate::core::source) | The [`Source`] trait: producers of raw bytes. |
//! | [`decoder`](crate::core::decoder) | The [`Decoder`] trait: bytes → typed messages. |
//! | [`encoder`](crate::core::encoder) | The [`Encode`] trait: messages → bytes. |
//! | [`sink`](crate::core::sink) | The [`Sink`] trait: terminal destinations. |
//! | [`preprocessor`](crate::core::preprocessor) | The [`PreProcessor`] trait: enrichment / transform. |
//! | [`placement`](crate::core::placement) | The [`Placement`] trait: routing decisions. |
//! | [`pipeline`](crate::core::pipeline) | The [`Pipeline`] trait: wiring stages together. |
//! | [`metrics`](crate::core::metrics) | The [`MetricsCollector`] trait: observability. |
//! | [`lifecycle`](crate::core::lifecycle) | Lifecycle phases for resource-owning stages. |
//! | [`error`](crate::core::error) | Explicit, typed, contextual errors. |
//!
//! ## Design rules
//!
//! - Keep `flyby::core` platform independent.
//! - No `unsafe` ever (enforced by `#![forbid(unsafe_code)]`).
//! - No backend-specific code leaks in here.
//! - Every public item is documented before stabilization.

#![deny(missing_docs)]
#![deny(rustdoc::broken_intra_doc_links)]

pub mod decoder;
pub mod encoder;
pub mod error;
pub mod lifecycle;
pub mod message;
pub mod metrics;
pub mod pipeline;
pub mod placement;
pub mod preprocessor;
pub mod sink;
pub mod source;

pub use decoder::Decoder;
pub use encoder::Encode;
pub use error::{Error, ErrorKind, Result};
pub use lifecycle::Lifecycle;
pub use message::{DefaultSchemaId, Message, Metadata, SchemaId, Timestamp};
pub use metrics::{CountingCollector, MetricKey, MetricKind, MetricsCollector, NullCollector};
pub use pipeline::{Pipeline, StepOutcome};
pub use placement::{Placement, SinkId};
pub use preprocessor::PreProcessor;
pub use sink::Sink;
pub use source::Source;

/// Prelude containing the core traits and error types.
///
/// Users should normally write:
///
/// ```rust
/// use flyby::core::prelude::*;
/// ```
pub mod prelude {
    pub use crate::core::decoder::Decoder;
    pub use crate::core::encoder::Encode;
    pub use crate::core::error::{Error, Result};
    pub use crate::core::lifecycle::Lifecycle;
    pub use crate::core::message::{DefaultSchemaId, Message, Metadata, SchemaId, Timestamp};
    pub use crate::core::metrics::{
        CountingCollector, MetricKey, MetricKind, MetricsCollector, NullCollector,
    };
    pub use crate::core::pipeline::{Pipeline, StepOutcome};
    pub use crate::core::placement::{Placement, SinkId};
    pub use crate::core::preprocessor::PreProcessor;
    pub use crate::core::sink::Sink;
    pub use crate::core::source::Source;
}