Crate meathook 

meathook

A polling runtime with composable, durable sinks: poll a source on an interval, buffer samples into time windows, and ship them to long-term storage — durably.

The core (traits, sink combinators, write-ahead spool, supervised runtime) is free of any HTTP/IO stack dependency: with --no-default-features you get just that and bring your own collector and terminal sink.

The two core abstractions are:

• Collector: produces a batch of records per tick. With the satay feature, satay::SatayCollector adapts any satay-generated API client into a collector.
• Sink: receives records. Sinks compose like tower layers — a buffering tier (Buffered), a durable write-ahead spool (DiskSpool), fan-out (Tee), and terminal sinks such as HfSink (feature huggingface) stack via SinkExt.

Pipelines (collector + sink stack) are supervised by the Meathook runtime: one tokio task each, respawn-on-panic with backoff, and a final flush() through the whole stack on graceful shutdown.

Re-exports

pub use collector::Collector;

pub use layer::Buffered;

pub use layer::DiskSpool;

pub use layer::FlushPolicy;

pub use layer::SinkExt;

pub use layer::SpoolError;

pub use layer::Tee;

pub use layer::TeeError;

pub use pipeline::Pipeline;

pub use runtime::Meathook;

pub use runtime::MeathookBuilder;

pub use runtime::RuntimeError;

pub use satay::SatayCollector;

pub use sink::Sink;

pub use sink::WindowMeta;

pub use encode::EncodeError;

pub use sink::huggingface::HfSink;

pub use sink::huggingface::HfSinkError;

Modules

collector

The Collector trait: a source that produces a batch of records per tick.

encode

Parquet encoding via serde_arrow: records stay plain #[derive(Serialize)] structs, no manual arrow builders.

layer

Sink combinators: the tower-layer part of meathook.

pipeline

Pipeline: one collector wired to one composed sink stack.

runtime

Meathook: the supervisor runtime.

satay

SatayCollector: optional adapter (feature satay) turning any satay-generated API client into a Collector.

sink

The Sink trait and window metadata passed alongside records.