ferrokinesis

A local AWS Kinesis mock server for testing, written in Rust.

Browser Demo

Try ferrokinesis without installing anything:

The GitHub Pages demo runs the in-process WASM wrapper entirely in your browser. It keeps state in memory only, so refresh or Reset state starts you from a clean slate.

Run it locally with:

npm ci --prefix demo
npm --prefix demo run dev

Features

• Pure Rust implementation
• Fully in-memory storage using DashMap + per-stream RwLock with lock-free per-shard sequence generation
• Implements all 39 Kinesis Data Streams API operations
• Supports both JSON and CBOR content types
• Health check endpoints (/_health, /_health/ready, /_health/live) for Docker/K8s
• TOML configuration file support (--config)
• Configurable AWS account ID, region, shard iterator TTL, and request body size limit
• Retention period enforcement with configurable TTL-based record trimming
• Graceful shutdown
• TLS support with built-in certificate generation
• 500+ tests (integration, property-based, unit) with multi-language SDK conformance tests (Go, Python, Node.js, Java v1/v2, KCL)
• Criterion benchmarks with CI regression gate (>10% threshold)

Installation

Binary

Download pre-built binaries from GitHub Releases:

# macOS (Apple Silicon)
curl -L https://github.com/mandrean/ferrokinesis/releases/latest/download/ferrokinesis-macos-arm64 -o ferrokinesis
chmod +x ferrokinesis

# macOS (Intel)
curl -L https://github.com/mandrean/ferrokinesis/releases/latest/download/ferrokinesis-macos-amd64 -o ferrokinesis
chmod +x ferrokinesis

# Linux (amd64)
curl -L https://github.com/mandrean/ferrokinesis/releases/latest/download/ferrokinesis-linux-amd64 -o ferrokinesis
chmod +x ferrokinesis

Cargo

cargo install ferrokinesis

WASI Preview 2 (Experimental)

Build the experimental WASI listener binary:

cargo build --target wasm32-wasip2 --no-default-features --features wasi --bin ferrokinesis-wasi

Then run it in a Preview 2-capable runtime with TCP listening enabled, for example:

wasmtime run --wasi tcp-listen=0.0.0.0:4567 target/wasm32-wasip2/debug/ferrokinesis-wasi.wasm

The WASI binary is env-configured only. It currently targets the normal JSON/CBOR request path and health endpoints; SubscribeToShard remains unsupported there for now.

Browser Demo

Build the browser demo and its generated WASM package:

npm ci --prefix demo
npm --prefix demo run build

Docker

docker run -p 4567:4567 ghcr.io/mandrean/ferrokinesis

Quick Start

Start the server:

docker run -p 4567:4567 ghcr.io/mandrean/ferrokinesis

Example using aws CLI:

# Create a stream with 2 shards
aws kinesis create-stream \
    --stream-name example-stream \
    --shard-count 2 \
    --endpoint-url http://localhost:4567 \
    --region us-east-1

# Publish a record
aws kinesis put-record \
    --stream-name example-stream \
    --partition-key pk1 \
    --data $(echo -n "hello world" | base64) \
    --endpoint-url http://localhost:4567 \
    --region us-east-1

See the full example with get-records and cleanup: examples/aws-cli/quickstart.sh

More examples using Rust, Python, Node.js, Go, and Java are available in the examples/ directory.

Usage

$ ferrokinesis --help

A local AWS Kinesis mock server for testing

Usage: ferrokinesis [OPTIONS]
       ferrokinesis <COMMAND>

Commands:
  serve         Start the mock Kinesis server (default when no subcommand is given)
  health-check  Run a health check against a running server (for Docker HEALTHCHECK)
  help          Print this message or the help of the given subcommand(s)

Options:
      --config <CONFIG>
          Path to a TOML configuration file [env: FERROKINESIS_CONFIG=]
      --port <PORT>
          The port to listen on [env: FERROKINESIS_PORT=]
      --account-id <ACCOUNT_ID>
          AWS account ID used in ARN generation (12-digit numeric) [env: AWS_ACCOUNT_ID=]
      --region <REGION>
          AWS region used in ARN generation and responses [env: AWS_REGION=]
      --create-stream-ms <CREATE_STREAM_MS>
          Amount of time streams stay in CREATING state (ms) [env: FERROKINESIS_CREATE_STREAM_MS=]
      --delete-stream-ms <DELETE_STREAM_MS>
          Amount of time streams stay in DELETING state (ms) [env: FERROKINESIS_DELETE_STREAM_MS=]
      --update-stream-ms <UPDATE_STREAM_MS>
          Amount of time streams stay in UPDATING state (ms) [env: FERROKINESIS_UPDATE_STREAM_MS=]
      --shard-limit <SHARD_LIMIT>
          Shard limit for error reporting [env: FERROKINESIS_SHARD_LIMIT=]
      --iterator-ttl-seconds <ITERATOR_TTL_SECONDS>
          Shard iterator time-to-live in seconds (minimum: 1, maximum: 86400)
          [env: FERROKINESIS_ITERATOR_TTL_SECONDS=]
      --max-request-body-mb <MAX_REQUEST_BODY_MB>
          Maximum request body size in megabytes (minimum: 1, maximum: 4096)
          [env: FERROKINESIS_MAX_REQUEST_BODY_MB=]
      --retention-check-interval-secs <RETENTION_CHECK_INTERVAL_SECS>
          Retention reaper interval in seconds (0 = disabled, maximum: 86400)
          [env: FERROKINESIS_RETENTION_CHECK_INTERVAL_SECS=]
  -h, --help
          Print help

Health Check Endpoints

Three endpoints are available for container orchestration and monitoring:

The built-in health-check subcommand can be used for Docker HEALTHCHECK:

HEALTHCHECK CMD ["ferrokinesis", "health-check"]

API & Test Coverage

39/39 operations implemented (100%)

TLS / HTTPS

ferrokinesis supports TLS when built with the tls feature:

Install with TLS support

cargo install ferrokinesis --features tls

Generate a self-signed certificate

ferrokinesis generate-cert
# Writes cert.pem and key.pem to the current directory

# Custom output paths and SANs:
ferrokinesis generate-cert \
    --cert-out /path/to/cert.pem \
    --key-out /path/to/key.pem \
    --san localhost --san 127.0.0.1 --san my-hostname

Start the server with TLS

ferrokinesis --tls-cert cert.pem --tls-key key.pem
# Listening at https://0.0.0.0:4567

Connecting with SDK clients

Runnable TLS examples for each SDK are in the examples/ directory:

Building

cargo build --release

Testing

cargo test

Benchmarking

See BENCHMARK.md for details on running micro-benchmarks and load tests.

Acknowledgements

Inspired by kinesalite by Michael Hart.

License

MIT