# Examples
Every example is a whole program: build it, run it, read what it printed. Each
one's header comment says what it demonstrates and what it needs, and the
tables below group them by what you are trying to do.
New here, read [Getting started](#1-getting-started) in order. Nothing outside
that tier depends on reading order — pick the one matching your task.
## Running one
```sh
cargo run -p spate --example memory_pipeline
cargo run --release -p spate --features full --example kafka_avro_to_clickhouse
```
An example whose **Features** column is not `—` needs those features on the
command line. Naming one without them fails with exactly what is missing:
```text
error: target `s3_backfill` in package `spate` requires the features: `s3`, `json`
```
Examples that talk to real servers ship no `docker-compose`. Each one's header
comment names exactly what it needs and which environment variables point at
it, and every shipped config reads its endpoints through `${VAR:-default}`, so
the file you run against your own servers is the file in this directory.
Configuration comes from `SPATE_CONFIG` where an example loads YAML from disk:
```sh
SPATE_CONFIG=/etc/spate/pipeline.yaml cargo run --release -p spate \
--features full --example kafka_avro_to_clickhouse
```
## The storefront stream
`spate-datagen` generates one dataset with nothing installed behind it: a
storefront whose payments and refunds name orders that were really placed, for
amounts matching their lines.
```text
order_placed { order_id, customer_id, region, placed_at, lines: [{ sku, qty, unit_cents }] }
payment_captured { order_id, amount_cents }
refund_issued { order_id, amount_cents, reason }
```
The nested `lines` array is what `flat_map` fans out, and the three event kinds
are what a split terminal separates. Sharding keys on the **order id**, which
the generator sets as each record's key: a payment and a refund carry only the
`order_id` of the order they settle, so that is the only field all three share
and the only one that can colocate them on a shard.
The types are `spate_datagen::storefront`, so an example and your own code can
share them.
## 1. Getting started
No infrastructure. Read them in this order.
| [`memory_pipeline`](memory_pipeline.rs) | How to build, drive and assert on a whole pipeline with **no infrastructure** | — | nothing |
| [`storefront_pipeline`](storefront_pipeline.rs) | How to run a realistic order stream to completion or to SIGTERM with **no infrastructure** | — | nothing |
| [`json_skip_bad_records`](json_skip_bad_records.rs) | How to decode many records from one NDJSON payload and skip malformed lines rather than stop with **JSON** | `json` | nothing |
| [`config_in_code`](config_in_code.rs) | How to configure a pipeline in code instead of loading a file with **no infrastructure** | — | nothing |
## 2. Production pipelines
The shapes a real deployment takes.
| [`avro_schema_evolution`](avro_schema_evolution.rs) | How to add a field to a producer's schema without breaking a running pipeline with **Avro** | `avro` | nothing |
| [`clickhouse_aggregating_mv`](clickhouse_aggregating_mv.rs) | How to roll orders up per region through a Null landing table into an AggregatingMergeTree with **ClickHouse** | `clickhouse` | ClickHouse |
| [`kafka_avro_flatmap_clickhouse`](kafka_avro_flatmap_clickhouse.rs) | How to fan an order's lines into a row each and shard them by order with **Kafka, Avro and ClickHouse Native** | `full` | Kafka and ClickHouse |
| [`kafka_avro_to_clickhouse`](kafka_avro_to_clickhouse.rs) | How to load an Avro order stream from Kafka into ClickHouse with **Kafka, Avro and ClickHouse** | `full` | Kafka, a schema registry and ClickHouse |
| [`kafka_to_kafka_split`](kafka_to_kafka_split.rs) | How to fan an order stream out to per-region topics with **Kafka** | `kafka` | Kafka |
| [`multi_table_split`](multi_table_split.rs) | How to route payments and refunds to a table each from one event stream with **Kafka and ClickHouse** | `full` | Kafka and ClickHouse |
## 3. Bounded jobs and scaling out
Work that finishes, and work shared across instances.
| [`nats_coordinated_backfill`](nats_coordinated_backfill.rs) | How to coordinate a fleet over the durable store with **NATS JetStream** | `s3,json,coordination-nats` | a NATS server with JetStream; run the binary twice |
| [`s3_backfill`](s3_backfill.rs) | How to backfill historical records from object storage and stop when the prefix is done with **object storage** | `s3,json` | nothing; it stages a `file://` bucket |
| [`s3_coordinated_backfill`](s3_coordinated_backfill.rs) | How to split one backfill across two instances without either duplicating it with **object storage and coordination** | `s3,json,coordination` | nothing |
## 4. Operating
What the pipeline tells you, and what it does when something breaks.
| [`custom_metrics`](custom_metrics.rs) | How to register your own metrics beside the framework's with **the Meter API** | — | nothing |
| [`instrumented_operator`](instrumented_operator.rs) | How to count what an operator you wrote is doing with **the Meter API** | — | nothing |
| [`sink_failures`](sink_failures.rs) | How to see what a failing sink does to the watermark with **no infrastructure** | — | nothing |
## 5. Extending
Writing your own components against the v1 contracts.
| [`custom_coordinated_source`](custom_coordinated_source.rs) | How to write a coordination-aware source from planner to driver with **the coordination seam** | `coordination` | nothing |
| [`custom_operator`](custom_operator.rs) | How to drive a chain of stateful operators over owned and borrowed records by hand with **no infrastructure** | — | nothing |
| [`custom_source_sink`](custom_source_sink.rs) | How to write a source, a payload-aware router and a sink from scratch with **no infrastructure** | — | nothing |
| [`manual_assembly`](manual_assembly.rs) | How to drop below the builder to the primitives it composes with **no infrastructure** | — | nothing |
## Containers
[`examples/docker`](https://github.com/spate-etl/spate/tree/main/examples/docker)
builds the flagship example into a distroless image, and its README covers
probes, drain timeouts and sizing.
## Related
- [User guide](https://spate.kainth.dev/docs/user-guide/) — the concepts these
examples are worked instances of
- [`spate` on docs.rs](https://docs.rs/spate) — every type and feature named here
- [`spate-test`](https://docs.rs/spate-test) — the in-memory source and sink the
infrastructure-free examples are built on