Crate arcly_stream

Expand description

arcly-stream — an open-extensible, high-performance live-media streaming kernel.

The pub/sub core of a streaming server: lock-free, zero-copy frame fan-out to an arbitrary number of late-joining subscribers, with instant-start GOP replay, publish/play authorization, live QoS, a pluggable multi-protocol ingestion layer (InboundProtocol), and a feature-gated pure-Rust media plane — all free of any config schema, metrics singleton, or HTTP runtime. You bring the wire protocols and telemetry by implementing small traits; the kernel owns the hard, hot-path part.

use arcly_stream::prelude::*;

let engine = Engine::builder()
    .max_publishers(1000)
    .application(AppSpec::new("live").gop_cache(120)) // instant-start
    .build();
assert_eq!(engine.list_apps().len(), 1);

§Architecture

The crate is layered as a small always-on kernel, a feature-gated media plane, and deferred wire protocols that exist only as traits:

  ┌─────────────────────── Core kernel (always on) ───────────────────────┐
  │  Engine · StreamHandle (broadcast fan-out, GOP cache, live QoS)        │
  │  PublishRegistry / PlaybackRegistry / EventBus · Observer · auth ·     │
  │  audit · health · MediaFrame / CodecId · StreamError                   │
  └───────────────────────────────────────────────────────────────────────┘
        ▲ implements                              ▲ feature-gated, pure Rust
        │                                         │
  ┌─────┴── Wire protocols & muxers ──┐   ┌───────┴── Media plane ──────────┐
  │ rtmp   RTMP publish/play (ready)  │   │ codec  H.264/H.265/AV1/VP9/VVC  │
  │ mpegts MPEG-TS muxer (ready)      │   │ hls    Packager + segmenter     │
  │ ProtocolHandler  (SRT/WHIP/RTSP)  │   │ record RecordingSink            │
  │ Transcoder/ClusterRelay (traits)  │   │ ingest TCP accept loop          │
  └───────────────────────────────────┘   └─────────────────────────────────┘

Two wire-plane components ship operational: the rtmp handler (publish + play over real RTMP) and the MpegTsMuxer (player-compatible .ts output). Remaining protocols (SRT/WHIP/RTSP) and ABR/cluster backends are still trait contracts a host implements.

§Design invariants

Zero-copy fan-out — frames are bytes::Bytes; one publish clones an Arc<MediaFrame> pointer to every subscriber, never the payload.
No locks on the hot path — the frame bus is a tokio::broadcast channel; cached CONFIG frames and the GOP head use ArcSwap / atomics.
No global state — telemetry (Observer), authorization (StreamAuthenticator), and audit (AuditSink) are injected traits with no-op/permit-all defaults, never singletons. The engine is Send + Sync and can be instantiated many times per process.
Contracts are traits — protocols depend on PublishRegistry / PlaybackRegistry, not on the concrete Engine, so a host can swap in its own bus.
Pluggable ingest — a new wire protocol is one InboundProtocol (inbound) implementation, registered with EngineBuilder::protocol. The engine runs any worker that satisfies the trait; the bundled RtmpHandler is just the first reference implementation.
#![forbid(unsafe_code)] — the kernel contains no unsafe, enforced by the compiler. Documentation completeness (#![deny(missing_docs)]) and intra-doc link integrity are compiler-enforced too.

§Module layout

Module	Responsibility
`frame`	`MediaFrame`, `CodecId`, `FrameType`, `FrameFlags`
`identity`	`AppName`, `StreamId`, `StreamKey` (cheap `Arc<str>` ids)
`error`	`StreamError` and the crate `Result` alias
`traits`	`MediaSource`, `MediaSink`, `ProtocolHandler`, `StorageBackend`, `HwAccelBackend`
`bus`	`StreamHandle` fan-out (GOP cache, live QoS) + registry contracts
`inbound`	`InboundProtocol` trait + `IngestContext` / `PublishSession` — the multi-protocol ingestion seam
`engine`	`Engine`, `EngineBuilder`, `AppSpec` — composition, run loop, source pump, idle reaper
`auth`	`StreamAuthenticator` publish/play authorization (permit-all default)
`observe`	`Observer` telemetry hook + `NoopObserver`
`audit`	`AuditSink` + `AuditPipeline` lifecycle audit trail
`health`	`HealthCheck` / `HealthRegistry` readiness aggregation
`transcode`	`Transcoder` / `RenditionSpec` ABR contracts
`cluster`	`ClusterRelay` origin/edge federation contracts
`testing`	In-memory helpers for downstream tests

§Cargo features

Feature	Effect
(none)	Pure engine: frame model, bus, traits, auth/audit/health/cluster contracts
`codec`	Baseline H.264 NAL/SPS + AAC ADTS parsers (`codec`)
`codec-h265` / `codec-av1` / `codec-vp9` / `codec-vvc`	Next-gen codec parsers (opt-in, pure Rust)
`codecs-all`	Every codec parser at once
`storage-fs`	Filesystem `StorageBackend` adapter
`ingest`	Shared TCP accept loop, keyframe gate, rate limiter, NAL scan
`hls`	HLS/LL-HLS packager: `Muxer`/`Packager`, segmenter, playlists (`packager`)
`mpegts`	Native MPEG-TS `Muxer` (`MpegTsMuxer`) — playable `.ts` segments
`fmp4`	Native fragmented-MP4/CMAF `Muxer` (`Fmp4Muxer`) — `ftyp`+`moov` init & `moof`+`mdat` fragments for LL-HLS
`rtmp`	Working RTMP publish/play `ProtocolHandler`
`record`	Segment/recording sink over a `StorageBackend` (`record`)
`hls` / `record`	Shared keyframe-boundary `segment` timing
`metrics`	Prometheus `Observer` implementation
`auth-token`	Production signed-token `StreamAuthenticator` (`TokenAuthenticator`) — HMAC-SHA-256, dependency-free
`macros`	`#[derive(MediaSink)]`, `#[protocol]` ergonomics
`full`	everything

Ready-to-use reference implementations ship for the operational traits too: FileAuditSink (AuditSink → file), StandardTelemetry (structured-logging Observer), and the metrics-gated PrometheusObserver.

Re-exports§

pub use audit::AuditAction;
pub use audit::AuditPipeline;
pub use audit::AuditRecord;
pub use audit::AuditSink;
pub use audit::FileAuditSink;
pub use auth::TokenAuthenticator;auth-token
pub use auth::AllowAll;
pub use auth::Credentials;
pub use auth::StreamAuthenticator;
pub use bus::EventBus;
pub use bus::PlaybackRegistry;
pub use bus::PublishRegistry;
pub use bus::Qos;
pub use bus::StreamEvent;
pub use bus::StreamEventKind;
pub use bus::StreamHandle;
pub use bus::StreamMetadata;
pub use bus::StreamState;
pub use bus::Subscription;
pub use cluster::ClusterRelay;
pub use cluster::NodeAddr;
pub use engine::AppSpec;
pub use engine::Engine;
pub use engine::EngineBuilder;
pub use engine::EngineConfig;
pub use error::ProtocolErrorKind;
pub use error::StreamError;
pub use frame::AudioFrame;
pub use frame::CodecId;
pub use frame::FrameFlags;
pub use frame::FrameType;
pub use frame::MediaFrame;
pub use frame::VideoFrame;
pub use health::HealthCheck;
pub use health::HealthRegistry;
pub use health::HealthReport;
pub use health::HealthStatus;
pub use identity::AppName;
pub use identity::StreamId;
pub use identity::StreamKey;
pub use inbound::InboundProtocol;
pub use inbound::IngestContext;
pub use inbound::PublishSession;
pub use observe::NoopObserver;
pub use observe::Observer;
pub use observe::StandardTelemetry;
pub use traits::HwAccelBackend;
pub use traits::MediaSink;
pub use traits::MediaSource;
pub use traits::ProtocolHandler;
pub use traits::StorageBackend;
pub use transcode::RenditionSpec;
pub use transcode::Transcoder;
pub use async_trait;
pub use bytes;

Modules§

audit: Compliance-grade audit trail for stream lifecycle.
auth: Stream-level authentication and authorization.
bus: The stream bus: the live pub/sub core, plus the trait contracts that decouple protocol/packager code from any concrete registry.
cluster: Multi-node clustering contracts: origin/edge discovery and stream relay.
codeccodec: Codec bitstream helpers — turning wire bytes into engine metadata and MediaFrame classification.
engine: The composition root — the analogue of arcly-http’s App.
error: The crate-wide error type.
frame: The media frame model — the unit of data that flows through the engine.
health: Liveness/readiness health checks.
identity: Cheaply-cloneable identifiers for applications and streams.
inbound: The multi-protocol ingestion architecture — the public seam for teaching the engine new inbound wire protocols (RTSP, SRT, WebRTC WHIP/WHEP, …) without touching the kernel.
observabilitymetrics: A Prometheus implementation of the Observer trait.
observe: Telemetry hook — the injected replacement for stream-center’s global Prometheus singleton (sc_metrics::Metrics::global()).
packagerhls: Egress packaging: turn a live frame stream into HLS/LL-HLS segments.
prelude: use arcly_stream::prelude::*; — everything a downstream user needs.
protocolingest: Reusable protocol scaffolding shared by ingest handlers (RTMP, RTSP, …).
recordrecord: Continuous recording: a MediaSink that persists frames through a StorageBackend.
segmenthls or record: Keyframe-boundary segmentation timing, shared by the HLS packager and the recorder.
storagestorage-fs: Reference StorageBackend implementations.
testing: In-memory helpers for exercising the engine in downstream tests.
traits: The foundational extension traits a downstream user implements.
transcode: Transcoding & adaptive-bitrate contracts.

Type Aliases§

Result: The crate-wide fallible result. Mirrors sc-core::Result.