Crate dspk_pipeline 

dspk pipelines as data: a versioned node-graph spec (PipelineSpec) and an executor (run) that streams a recording through dspk-ops stages on the device.

{"dspk_pipeline": 1,
 "nodes": {"raw": {"op": "read", "path": "recording.bin"},
           "hp":  {"op": "iir", "band": "highpass", "cutoff": [300.0], "mode": "forward"},
           "out": {"op": "write", "path": "filtered.zarr", "format": "zarr"}},
 "edges": [["raw", "hp"], ["hp", "out"]]}

• JSON is canonical; TOML is accepted for hand-written files (PipelineSpec::load); the JSON Schema of the spec is PipelineSpec::json_schema.
• The executor reads windows with read-ahead, uploads each window once (integer recordings as stored integers, scaled on the device), runs every stage on the device, downloads once and hands the result to a writer thread (raw, or Zarr chunks directly); the canonical spec is written next to the output as provenance (<output>.pipeline.json). The report counts the bytes moved at each step and the time spent in each (StageTimings).
• One spec runs over many inputs (globs, lists) and may read several recordings as one (merge).
• plan says what a run would do (mode, halos, windows, bytes) without touching the device; run_with reports progress after each window and can cancel.

Structs

Chain

The chain of a valid spec, in order: the read node, the processing nodes, the write node.

DataMovement

Bytes moved by a run, step by step (window halos included: they are read, uploaded and downloaded with their window).

Execution

How the executor runs the chain.

MergePart

One recording of a merge.

PipelineSpec

A pipeline: operations (nodes, by id) and the edges between them.

Plan

What running one input would do, without touching the device.

Progress

Where a run is, passed to the observer after each window.

RunReport

What a run did.

StageTimings

Where a run spent its time, in seconds. Reading, the device and the writer run on three threads at once, so these add up to more than RunReport::seconds; the largest of read_wait, device and encode + disk is the bottleneck.

WhitenFit

How a whiten node is fitted (its spec fields).

Enums

Band

Compression

Compression of Zarr chunks: {"codec": "zstd", "level": 3}, {"codec": "blosc_lz4", "level": 5} (LZ4 after a byte shuffle; faster, usually a little larger) or {"codec": "none"}.

Design

Direction

Which way a crossings node’s signal goes past its level: rising (up through a level above the baseline), falling (down through a level below it) or both (past +|level| upwards or −|level| downwards; an event’s sign is its peak’s).

Edge

Samples past the ends of a window (scipy names: constant 0, odd, reflect, nearest).

ExecMode

stream: windows in order, filter state carried (exact; forward filters, point-wise and spatial stages only). windows: each window on its own with halos of the chain’s settling (zero-phase filters, stencil stages). auto: stream when the chain allows it.

ExecutedMode

How the chain runs.

Input

One input of a read node: a path (or glob), or several recordings merged side by side.

Mode

forward_backward: zero phase (sosfiltfilt); forward: causal (sosfilt).

Node

One operation. read and write are the ends; the others mirror dspk.ops.stage.

OutputFormat

PipelineError

Why a pipeline could not be read, checked or run.

Start

How a forward filter starts: at rest, or from the steady state of the first sample.

Upload

What crosses to the device. stored: the recording’s stored integers, scaled on the device (int16 moves half the bytes of float32). scaled: float32 values scaled on the host. auto: stored for integer recordings, scaled otherwise (float32 is the same either way; float64 is scaled on the host).

Constants

DEFAULT_BLOSC_LEVEL

Default LZ4 level of Compression::BloscLz4 (blosc’s 0–9).

DEFAULT_CHUNK_SAMPLES

Samples per channel in each window the executor processes (2.2 s at 30 kHz).

DEFAULT_ZSTD_LEVEL

Default zstd level of Compression::Zstd.

EVENTS_VERSION

Format version of an events Zarr group (dspk_events attribute).

PROVENANCE_SUFFIX

Appended to the output path for the canonical spec written next to it (provenance).

SPEC_VERSION

The spec version this crate reads and writes (dspk_pipeline field).

WHITEN_DEFAULT_CHUNKS

Chunks those seconds are split into, by default.

WHITEN_DEFAULT_FIT_S

Seconds a whiten node fits on, by default.

WRITE_QUEUE_WINDOWS

Downloaded windows waiting for the writer thread, at most: enough to absorb a slow write, and a bound on the memory they hold (each is channels × window float32 values).

Functions

default_open

Opens a recording with dspk_io (generic formats): source picks a source within the file.

events_csv

The events table as CSV (the columns of events_frame).

events_frame

The events of a crossings chain as a table, one row per event (see OutputFormat::Events): channel (u32), channel_name, onset_sample / offset_sample / peak_sample (u64; offset null when the range ended inside the event), peak_value (f32), onset_s / offset_s / duration_s (f64).

fit_whitening

Fits a global ZCA whitening of channels of source as processed by before (the stages ahead of it): fit.chunks chunks of fit.fit_s / fit.chunks seconds spread evenly over the recording, each processed with the stages’ settling context, their covariance whitened.

plan

What crate::run would do for each input of spec, opening the inputs (metadata reads only) with dspk_io::open and never touching the device.

plan_with

plan with recordings opened by open.

read_events_zarr

The events of a Zarr group written by write_events_zarr, with its sample rate and channel names.

run

Runs spec once per input, resolving relative paths against base_dir, opening recordings with dspk_io::open (generic formats). One report per input, in order.

run_with

run with recordings opened by open (e.g. a crate that adds formats, such as dspk_synapse::io) and observe called after each window: returning false cancels the run (PipelineError::Cancelled; the unfinished output is removed, finished inputs stay).

unfitted_whitening

The identity on channels channels: a whitening’s place in a pipeline before it is fitted (same channels, no settling), so a run can be planned without the device.

write_events

Writes the events at path: a Zarr group when it ends in .zarr or lies inside a .zarr store (e.g. rec.dspk.zarr/events/biceps; see write_events_zarr), CSV when it ends in .csv, Parquet otherwise. Returns the bytes written (Zarr: the values).

write_events_zarr

Writes the events as a Zarr group rooted at directory dir (replacing it), the layout of dspk’s Zarr outputs: arrays channel (int32), onset_sample, peak_sample (uint64), offset_sample (int64, −1 for an event still open at the end) and peak_value (float32), one entry per event along events; attributes dspk_events (version), sample_rate_hz and channel_names. A group inside another store (e.g. rec.dspk.zarr/events/<name>) is written the same way.