Crate meathook 

meathook

A polling runtime with composable, durable sinks: poll a source on an interval, buffer samples into time windows, and ship them to long-term storage — durably.

The core (traits, sink combinators, write-ahead spool, supervised runtime) is free of any HTTP/IO stack dependency: with --no-default-features you get just that and bring your own collector and terminal sink.

The two core abstractions are:

• Collector: produces a batch of records per tick. With the satay feature, satay::SatayCollector adapts any satay-generated API client into a collector.
• Sink: receives records. SinkStack composes any number of buffering tiers (Tier), each backed by a pluggable Store: in-memory MemStore, durable write-ahead JsonlStore, or your own. Finish the stack with a terminal sink such as HfSink (feature huggingface); fan out completed sinks with SinkExt::tee.

Pipelines (collector + sink stack) are supervised by the Meathook runtime: one tokio task each, respawn-on-panic with backoff, and a final flush() through the whole stack on graceful shutdown.

Re-exports

pub use collector::Collector;

pub use layer::FlushPolicy;

pub use layer::SinkExt;

pub use layer::SinkStack;

pub use layer::Tee;

pub use layer::TeeError;

pub use layer::Tier;

pub use layer::TierError;

pub use pipeline::Pipeline;

pub use runtime::Meathook;

pub use runtime::MeathookBuilder;

pub use runtime::RuntimeError;

pub use satay::SatayCollector;

pub use sink::Sink;

pub use sink::WindowMeta;

pub use store::JsonlStore;

pub use store::JsonlStoreError;

pub use store::MemStore;

pub use store::Segment;

pub use store::Store;

pub use encode::Encoder;

pub use encode::JsonEncoder;

pub use encode::ParquetCompression;

pub use encode::ParquetEncodeError;

pub use encode::ParquetEncoder;

pub use encode::Uncompressed;

pub use encode::Zstd;

pub use sink::huggingface::CommitGate;

pub use sink::huggingface::HfSink;

pub use sink::huggingface::HfSinkError;

Modules

collector

The Collector trait: a source that produces a batch of records per tick.

encode

Pluggable window encoders: an Encoder turns one window of records into the bytes of a single file. Records stay plain #[derive(Serialize)] structs — no format-specific builders — and a terminal sink’s wire format is swapped by choosing an encoder. ParquetEncoder (feature parquet) stays the HfSink default; JsonEncoder is always available; [CsvEncoder] is gated behind the csv feature.

layer

Sink combinators: the tower-layer part of meathook.

pipeline

Pipeline: one collector wired to one composed sink stack.

runtime

Meathook: the supervisor runtime.

satay

SatayCollector: optional adapter (feature satay) turning any satay-generated API client into a Collector.

sink

The Sink trait and window metadata passed alongside records.

store

The Store trait: pluggable window-keyed storage backing a Tier layer.