flyby 0.1.1

A high-performance Rust framework for composable data-ingestion pipelines.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

#![doc = include_str!("../docs/README.md")]
//!
//! # flyby
//!
//! High-performance composable data-ingestion pipelines.
//!
//! ```rust
//! use flyby::prelude::*;
//! ```
//!
//! ## Crate layout
//!
//! | Module | Role |
//! |--------|------|
//! | [`core`] / [`api`] | Traits, errors, lifecycle, metrics |
//! | [`memory`] | Shared-memory SPSC sink (feature `memory`, default) |
//! | [`net`] | Network simulator + AF_XDP/DPDK stubs |
//! | [`storage`] | File source, framing, replay + io_uring/SPDK stubs |
//! | [`pipeline`] / [`runtime`] / [`builder`] | Composition and execution |
//!
//! ## Feature flags
//!
//! | Feature       | Default | Description                                  |
//! |---------------|---------|----------------------------------------------|
//! | `memory`      | yes     | In-process shared-memory sink.               |
//! | `af_xdp`      | no      | AF_XDP source stub (Linux eBPF / XSK).       |
//! | `dpdk`        | no      | DPDK source stub.                            |
//! | `io_uring`    | no      | io_uring storage backend stub.               |
//! | `spdk`        | no      | SPDK storage backend stub.                   |
//! | `simulator`   | no      | Builder flag for the in-process net simulator. |
//! | `benchmarks`  | no      | Reserved for optional bench wiring.          |
//!
//! ## Runtime (Part VII)
//!
//! See [`runtime`] for scheduling, back-pressure, configuration, and the
//! lifecycle driver. ADRs: batch-oriented runtime (009), runtime independent
//! of backends (010).

#![deny(missing_docs)]
#![deny(rustdoc::broken_intra_doc_links)]

/// Contract layer: traits, errors, lifecycle, and metrics.
pub mod core;

/// Shared-memory sink (default backend).
#[cfg(feature = "memory")]
pub mod memory;

/// Networking backends (simulator always; AF_XDP/DPDK behind features).
pub mod net;

/// Storage backends (file/framing/replay always; io_uring/SPDK behind features).
pub mod storage;

/// Core traits, errors, and lifecycle (alias of [`core`]).
pub mod api {
    pub use crate::core::{
        CountingCollector, Decoder, DefaultSchemaId, Encode, Error, ErrorKind, Lifecycle, Message,
        Metadata, MetricKey, MetricKind, MetricsCollector, NullCollector, Pipeline, Placement,
        PreProcessor, Result, SchemaId, Sink, SinkId, Source, StepOutcome, Timestamp,
    };
}

pub use api::*;

pub mod builder;
pub mod pipeline;
pub mod runtime;

/// The public prelude.
pub mod prelude {
    pub use crate::api::*;
    pub use crate::builder::{FlyBy, FlyByBuilder};
    pub use crate::pipeline::{
        CallbackPlacement, DropAllPlacement, FixedPlacement, HashPlacement, IdentityPreProcessor,
        NetworkBatchSource, RawBatchSource, RoundRobinPlacement, SimplePipeline,
        StorageBatchSource, schema_hash_placement,
    };
    pub use crate::runtime::{
        BackpressureStrategy, Runtime, RuntimeConfig, RuntimePhase, SchedulerKind,
        SingleThreadScheduler, WorkerPoolScheduler,
    };
}