tsink 0.10.1

Why tsink?

• Three deployment modes — embed the library directly in your Rust or Python application, run a standalone server binary, or form a replicated cluster. Same engine everywhere.
• Robust engine — segmented WAL with crash-safe sync, LSM-style leveled compaction, adaptive delta/XOR/zstd encoding, mmap zero-copy reads, and configurable memory backpressure.
• Tiered storage — hot, warm, and cold tiers with automatic lifecycle management and optional object-store backing.
• Drop-in protocol support — accepts Prometheus remote write/read, InfluxDB line protocol, OTLP, StatsD, and Graphite out of the box.
• Built-in PromQL — query your data with a native PromQL parser and evaluator. No external query layer needed.
• Secure by default — TLS (rustls, no OpenSSL), RBAC with OIDC, multi-tenant isolation, and mTLS between cluster nodes.
• Zero external dependencies at runtime — single static binary for the server; protoc is vendored at build time.

Deployment modes

Embedded library

Add tsink as a dependency and get a full time-series engine in-process — WAL durability, compaction, retention, and queries included.

use tsink::{DataPoint, Row, StorageBuilder, TimestampPrecision};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let storage = StorageBuilder::new()
        .with_data_path("./tsink-data")
        .with_timestamp_precision(TimestampPrecision::Milliseconds)
        .build()?;

    storage.insert_rows(&[
        Row::new("cpu_usage", DataPoint::new(1_700_000_000_000_i64, 42.0)),
    ])?;

    let points = storage.select("cpu_usage", &[], 1_700_000_000_000, 1_700_000_000_001)?;
    println!("{points:?}");

    storage.close()?;
    Ok(())
}

UniFFI bindings expose the core API as a native Python module:

from tsink import TsinkStorageBuilder, DataPoint, Row, Value

builder = TsinkStorageBuilder()
builder.with_data_path("./tsink-data")
db = builder.build()

db.insert_rows([
    Row(
        metric="cpu_usage",
        labels=[],
        data_point=DataPoint(timestamp=1_700_000_000_000, value=Value.F64(v=42.0)),
    )
])
print(db.select("cpu_usage", [], 0, 2_000_000_000_000))

Server mode

A single binary that speaks every major metrics protocol.

cargo run -p tsink-server --bin tsink-server --release -- \
  --listen 127.0.0.1:9201 \
  --data-path ./var/tsink

Write data with any client you already have:

# Prometheus text exposition
curl -X POST http://127.0.0.1:9201/api/v1/import/prometheus \
  -H 'Content-Type: text/plain' \
  -d 'http_requests_total{method="GET"} 1027 1700000000000'

# PromQL query
curl 'http://127.0.0.1:9201/api/v1/query?query=http_requests_total'

Cluster mode

Enable clustering with a flag and scale horizontally. tsink handles shard routing, replication, consistency, hinted handoff, repair, and rebalance automatically.

tsink-server \
  --listen 0.0.0.0:9201 \
  --data-path ./var/tsink \
  --cluster-enabled \
  --cluster-replication-factor 3 \
  --cluster-peers node-2:9202,node-3:9203

Storage engine

Ingestion protocols

Clustering & replication

• Consistent hash-ring sharding with configurable shard count
• Tunable replication factor and consistency levels (One / Quorum / All) for writes and reads
• Node roles — dedicated Storage, Query, or Hybrid nodes
• Hinted handoff — queues writes for temporarily unavailable replicas
• Digest-based repair — fingerprint exchange detects and resolves inconsistencies
• Online rebalance — pause, resume, and monitor shard migration
• Distributed query fan-out — concurrent shard-aware reads with merge limits
• Cluster-wide snapshots — coordinated data + control-plane backup and restore
• Internal mTLS — dedicated CA for peer-to-peer traffic

Security & multi-tenancy

• TLS — rustls-based with hot-reloadable certificates
• Authentication — bearer tokens (file or exec-based loading), OIDC JWT validation (RS256, HS256)
• RBAC — roles, service accounts with rotation, and live audit logging
• Multi-tenant isolation — per-tenant policies for write rate, query concurrency, admission budgets, and retention
• Secret rotation — runtime rotation of auth tokens, TLS certs, and mTLS materials with overlap grace periods

Operations

• /healthz and /ready — Kubernetes-compatible probes
• /metrics — Prometheus-format self-instrumentation
• Recording & alerting rules — built-in rules engine with configurable evaluation intervals
• Rollup policies — persistent downsampled materialization with automated scheduling
• Migration tooling — backfill, verify, and cutover from Prometheus, VictoriaMetrics, InfluxDB, OTLP, StatsD, and Graphite
• Support bundles — bounded JSON diagnostic snapshots per tenant

Documentation

Getting started

• Embedded library guide — using tsink as a Rust dependency, StorageBuilder configuration, sync and async APIs, snapshots
• Python bindings guide — UniFFI setup, TsinkStorageBuilder, type mappings, error handling
• Server deployment — running the single-node server binary, CLI flags, environment variables
• Cluster setup — multi-node deployment, peer discovery, shard count, replication factor, consistency levels, node roles

Architecture & design

• Architecture overview — high-level system design, component interactions, data flow
• Storage engine internals — WAL, segments, LSM-style compaction, encoding codecs, mmap reads, write buffer
• PromQL implementation — lexer, parser, evaluator, supported functions, aggregations, subqueries
• Clustering internals — consistent hash ring, replication protocol, hinted handoff, digest repair, rebalance, distributed queries

API & protocol reference

• HTTP API reference — all endpoints, request/response formats, authentication headers, error codes
• PromQL reference — function catalogue, operators, vector matching, type coercion rules
• Ingestion protocols — Prometheus remote write, InfluxDB line protocol, OTLP, StatsD, Graphite wire formats and endpoints
• Configuration reference — complete list of server, engine, cluster, and security options with defaults

Features

• Tiered storage — hot/warm/cold lifecycle, retention windows, object-store backing
• Compaction — L0/L1/L2 levels, merge strategies, tombstone handling, tuning
• Rollups & downsampling — rollup policies, materialization scheduling, query integration
• Data types & native histograms — float64, bytes, native histograms, timestamp precision modes
• Exemplars — exemplar storage, querying, cardinality limits

Security & operations

• Security model — TLS/mTLS setup, RBAC roles, OIDC authentication, audit logging
• Multi-tenancy — tenant isolation, per-tenant quotas, admission budgets, usage accounting
• Secret rotation — rotating auth tokens, TLS certificates, mTLS materials, grace periods
• Monitoring & observability — /metrics endpoint, self-instrumentation, health probes, support bundles
• Recording & alerting rules — rule definitions, evaluation intervals, recording rule output
• Performance tuning — memory budgets, compaction tuning, write pipelining, cgroup-aware scheduling
• Migration guide — migrating from Prometheus, VictoriaMetrics, InfluxDB; backfill, verify, cutover

License

MIT — see LICENSE.