Expand description
Signal-native logs, traces, and metrics storage for shard-stream.
A ShardTelemetry owns one single-writer LogStripe per shard-stream
physical shard. Call ShardTelemetry::apply_durable from the owning
shard-stream worker only after the associated append is durable. This keeps
the append log authoritative while allowing term and metadata indexes to be
queried through a per-partition indexed watermark.
Structs§
- Analytics
Row - One normalized row shared by the relation-specific Arrow writers.
- Analytics
Scan Request - Bounded storage-level scan requested by ClickHouse.
- Attribute
Fingerprint - Stable 128-bit identity of one exact typed metadata key/value.
- Block
Catalog - In-memory directory of sealed blocks owned by one log stripe.
- Block
Descriptor - Immutable metadata describing a sealed compression block.
- BlockId
- Monotonically assigned identifier for a sealed log data block.
- Block
Query Index - Exact term and metadata postings for one structural block.
- Cached
Object Range - One immutable object range backed by a shared verified cache chunk.
- Catalog
Group Entry - Bounded catalog entry pointing to one immutable group manifest.
- Catalog
Lease - Read lease for one immutable catalog generation.
- Catalog
Page - Immutable bounded page of group catalog entries.
- Catalog
Page Ref - Root-level coarse bounds and immutable pointer for one catalog page.
- Catalog
Pointer - Small mutable pointer atomically selecting the authoritative catalog root.
- Catalog
Root - Immutable catalog root for one physical-shard logical-partition pair.
- Compression
Block Assignment - Final assignment of one complete sub-block.
- Compression
Block Collator - Fixed-capacity, owner-local block collator.
- Compression
Block Score - Integer score for a complete block or sub-block.
- Compression
Cohort Id - Stable identifier for a compression cohort, such as a service and schema version.
- Compression
Locality Config - Bounded settings for one stripe-local block collator.
- Compression
Locality Record - One record presented to block collation.
- Compression
Locality Stats - Cumulative diagnostics for one stripe-local collator.
- Compression
Placement - Final owner-local placement decision for one collated block.
- Compression
Placement Id - Stable identifier for a final compression placement.
- Compression
Shard Profile - Read-only diagnostics for one active compression shard.
- Compression
Temperature - Integer compression-locality temperature.
- Correlation
Block Filter - Compact immutable filter used to prune cold signal blocks by shared telemetry identity. False positives are possible; false negatives are not.
- Correlation
Config - Hard bounds for one owner stripe’s cross-signal postings.
- Correlation
Index - Preallocated, single-writer correlation postings owned by one stripe.
- Correlation
Query - Cross-signal lookup constraints combined with AND semantics.
- Correlation
Stats - Snapshot of bounded correlation-index state.
- Decoded
Remote Write - A completely decoded and validated Remote Write request.
- Decoded
Structural Record - A record reconstructed from the single current structural block layout.
- Delete
Request - One durable Loki-compatible logical deletion request.
- Dictionary
Cache - Byte-bounded LRU for immutable compression dictionaries.
- Dictionary
Catalog - Control-plane catalog for immutable dictionaries and placement assignments.
- Dictionary
Catalog Snapshot - Immutable control-plane view of published compression dictionaries.
- Dictionary
Id - Stable identifier for an immutable compression dictionary.
- Dictionary
Insert - Outcome of inserting an immutable compression dictionary into a stripe cache.
- Dictionary
Publication - Result of atomically publishing an immutable dictionary assignment.
- Docker
LogRecord - Minimal owned Docker record used by the native ingestion fast path.
- Durable
Log - One log event accepted after the corresponding shard-stream append is durable.
- Durable
Metric Point - One durable raw metric point. Exactly one point consumes one logical offset.
- Durable
Span - One durable OTLP span. Exactly one instance consumes one logical offset.
- Durable
Telemetry Config - Standalone durable ShardTelemetry configuration.
- Durable
Telemetry Limits - Bounded memory, journal, and SSD-cache limits for one local durable store.
- Durable
Telemetry Store - Signal-native store whose acknowledged writes are durable shard-stream appends and whose reads execute on the owning ShardTelemetry stripe workers.
- Embedded
Frame Index - Lossless compressed-domain candidate index embedded in one structural frame.
- Embedded
Telemetry Config - Lifecycle configuration for a one-process embedded ShardTelemetry runtime.
- Embedded
Telemetry Runtime - One-owner embedded runtime suitable for a local node telemetry deployment.
- Explicit
Histogram Value - Explicit histogram point payload.
- Exponential
Histogram Buckets - Positive or negative exponential-histogram buckets.
- Exponential
Histogram Value - Exponential histogram point payload.
- Fast
Telemetry Config - Bounded configuration for a periodic
fast-telemetrysnapshot. - Fast
Telemetry Export Report - Counters returned by one embedded snapshot export.
- Fast
Telemetry Exporter - Background-driven bridge from a
fast-telemetryruntime to local durable storage. - Fetched
Telemetry Batch - One source batch recovered from the local WAL for durable upstream offload.
- Histogram
Bucket Span - One sparse native-histogram bucket span.
- Index
Receipt - Result of publishing one durable append into the hot index.
- Indexed
Structural Block - Structural bytes and their already-built compressed-domain index.
- Lifetime
Metric Rollup - Queryable outcome retained locally after raw metric points are removed or archived.
- Lifetime
Rollup Report - Result of one durable local lifetime-rollup checkpoint.
- Local
Object Store - Filesystem implementation of
TelemetryObjectStoreused for local operation and deterministic testing of S3-style immutable publication. - LogMatch
- A visible log record returned by a
LogQuery. - LogQuery
- Boolean log lookup against one logical partition.
- LogRegex
- Validated regular expression used by a log predicate.
- LogStripe
- A lock-free-by-ownership hot index for one shard-stream physical shard.
- Loki
ApiConfig - Configuration for the Loki-compatible HTTP boundary.
- Loki
ApiError - HTTP-boundary failure with a stable response status and safe message.
- Loki
ApiStore - Thread-safe in-memory reference backend used by differential API tests.
- Loki
Entry - One normalized Loki entry accepted by the compatibility boundary.
- Message
Fingerprint - Allocation-free message analysis used by routing and term indexing.
- Metadata
Field - A normalized metadata field stored alongside a log message.
- Metric
Exemplar - One metric exemplar nested under a point.
- Metric
Identity - Canonical identity of one metric series.
- Metric
Query - Native metric selector used by PromQL storage scans and direct APIs.
- Metric
Stripe - Bounded single-writer metric head and immutable raw chunk collection.
- Native
Capabilities - Capabilities negotiated before a remote producer is allowed to mark its telemetry integration ready.
- Native
Client Config - Connection and bounded-request settings for
ShardTelemetryClient. - Native
Client Error - Error returned by the reusable native client.
- Native
Frame - One complete native wire frame.
- Native
Frame Header - Validated fixed header for one multiplexed native frame.
- Native
LogQuery Result - Decoded stream-grouped native log query result.
- Native
Partition Ack - Per-partition acknowledgement returned by native protocol v1.
- Native
Partition Append - One routed STEL envelope in a signal-aware native append.
- Native
Protocol Error - Protocol validation error suitable for a native bad-request response.
- Native
Query - Bounded native indexed-query request.
- Native
Server Config - Runtime limits for the native TCP listener.
- Native
Telemetry Append Ack - Atomic native v1 response containing one acknowledgement per partition.
- Native
Telemetry Batch - Native protocol v1 append containing one envelope per resulting partition.
- Object
Metadata - Metadata returned for one object-store object.
- Object
Store Stats - Process-local object-tier operation counters shared by all stripe owners.
- Object
Tier Config - Bounded object-tier catalog configuration.
- Offload
Checkpoint - Durable state for one upstream-offloaded local WAL partition.
- Offload
Error - Error returned by the background store-and-forward pipeline.
- Offload
Loop Report - Cumulative outcome returned when a background offload loop is stopped.
- Offload
Report - Per-round store-and-forward outcome.
- Otlp
Ingest Service - Shared OTLP transport implementation backed by the durable telemetry store.
- Otlp
LogDecoder - Decoder for binary OTLP
ExportLogsServiceRequestpayloads. - Otlp
LogEvent - One normalized OpenTelemetry log event before shard-stream assigns its offset.
- Otlp
Metric Event - One fully validated OTLP metric point awaiting its durable offset.
- Otlp
Receiver Config - Production OTLP receiver limits and single-tenant identity.
- Otlp
Sink Config - Configuration for shard-telemetry’s per-shard native and OTLP index sinks.
- Otlp
Span Event - One fully validated OTLP span awaiting its durable offset.
- Otlp
Telemetry Decoder - Transport-independent OTLP trace and metric decoder.
- Persistent
Query Index - Immutable query directory for a set of sealed structural blocks.
- Production
Metrics Snapshot - Monotonic protocol counters exposed by the Prometheus endpoint.
- Production
Runtime - Shared single-tenant admission and lifecycle state for every protocol.
- Prometheus
ApiConfig - Single-tenant Prometheus compatibility limits.
- Prometheus
Service - Shared Prometheus protocol service backed by signal-native metric stripes.
- Promql
Engine - Rust PromQL evaluator backed by ShardTelemetry metric stripes.
- Promql
Error - Parse or evaluation error returned through the Prometheus API envelope.
- Promql
Limits - Limits for one native PromQL evaluation.
- Promql
Sample - One Prometheus float sample with its complete label set.
- Promql
Series - One matrix series returned by a range query.
- Query
Block Metadata - Offset and timestamp bounds for one independently compressed block.
- Query
Cursor - Exclusive continuation point for deterministic lookup pagination.
- Query
Hit - Exact location of a candidate record inside a sealed block.
- Realtime
Dictionary Config - Bounded policy for continuously training immutable Zstandard dictionaries.
- Realtime
Dictionary Observer - Cloneable, non-blocking block-level submission handle for log stripes.
- Realtime
Dictionary Stats - Snapshot of online dictionary-learning activity.
- Realtime
Dictionary Trainer - Background real-time dictionary learner and immutable publication service.
- Remote
Write Decoder - Lossless decoder for Remote Write v1 and v2 protobuf messages.
- Remote
Write Stats - Counts reported in the mandatory Remote Write 2.0 response headers.
- Resource
Context - Exact resource context shared by records from one OTLP resource group.
- Resource
Context Id - Stable 128-bit identity of an exact resource context.
- Retention
Report - Result of one batch-aligned physical retention pass.
- Retired
Object - One exact object whose previous catalog generation relinquished ownership.
- S3Object
Store - Rust-native S3 adapter with conditional publication and streaming uploads.
- S3Object
Store Config - Production S3 and S3-compatible object-store configuration.
- Scope
Context - Exact instrumentation scope context shared by records from one OTLP scope.
- Scope
Context Id - Stable 128-bit identity of an exact instrumentation scope context.
- Series
Accumulator Checkpoint - Checkpointed delta-to-cumulative state for PromQL views.
- Series
Fingerprint - Stable 128-bit identity of a metric series.
- Service
Lifecycle - Shared lifecycle controller for HTTP, native TCP, and process shutdown.
- Shard
Telemetry - Container for independently owned ShardTelemetry log stripes.
- Shard
Telemetry Client - Persistent, bounded native-v1 client for ShardTelemetry.
- Shard
Telemetry Config - Complete single-node ShardTelemetry configuration.
- Shared
Telemetry Object Store - Cloneable type-erased object-store handle used by production stripe owners.
- Signal
Config - Bounded configuration shared by a single telemetry signal.
- Signal
Tier Payload - One already encoded trace block or metric chunk awaiting immutable publication.
- Single
Tenant Config - Fail-closed limits and credentials for one production tenant.
- Sink
Object Tier Config - Immutable object-tier and bounded SSD-cache settings shared by sink stripes.
- Span
Event - One nested span event. It does not consume a shard-stream offset.
- SpanId
- A validated 64-bit OpenTelemetry span ID.
- Span
Link - One nested link to another span. It does not consume a shard-stream offset.
- Span
Status - Final OpenTelemetry span status.
- SsdCache
Config - Configuration for one byte-bounded SSD object-range cache.
- SsdCache
Stats - Runtime diagnostics for one recoverable SSD object cache.
- SsdObject
Cache - Recoverable, integrity-checked SSD cache for immutable object ranges.
- Store
Health - Health snapshot supplied by a Loki storage backend.
- Store
Metrics - Storage counters rendered with protocol counters by
/metrics. - Stripe
Config - Resource limits for one shard-aligned log stripe.
- Structural
LogMetadata Ref - Borrowed exact OTLP metadata exposed to the structural encoder.
- Summary
Quantile Value - One legacy summary quantile.
- Summary
Value - Legacy summary point payload.
- Telemetry
Attribute - One exact OTLP key/value pair.
- Telemetry
Entity Ref - An OTLP Resource entity reference.
- Telemetry
Envelope - One checksummed durable telemetry append payload.
- Telemetry
Host Attachment - Externally owned shard-stream engine and signal-service attachment.
- Telemetry
Object Tier - Partition-scoped immutable object tier with a conditionally published root.
- Telemetry
Record Ref - Stable address of one log record in shard-stream’s ordered append log.
- Telemetry
Router - Deterministic signal-aware logical partition router.
- Telemetry
Service - Cloneable read service for the stripes owned by durable sink workers.
- Telemetry
Sink Factory - Builds one owner-only log index worker for every shard-stream physical shard.
- Tempo
ApiConfig - Single-tenant Tempo compatibility limits.
- Tempo
Service - Shared Tempo-compatible query service.
- Text
Matcher - Reusable literal text matcher.
- Tier
Artifact - Immutable object metadata for one published group artifact.
- Tier
Artifact Source - Local source file to publish as one immutable group artifact.
- Tier
Block Entry - Durable block metadata and payload extent inside a block-group pack.
- Tier
Checkpoint - Durable sink watermark covered by an immutable object-tier group.
- Tier
Group Manifest - Immutable manifest for one independently queryable block group.
- Tier
Group Source - Complete local input needed to publish one block group.
- Tier
Query Range - Coarse inclusive bounds used to prune catalog pages and groups.
- Tier
Retention Report - Result of one bounded exact-key object-tier retention transaction.
- Trace
Directory - In-memory view of the immutable trace directory.
- TraceId
- A validated 128-bit OpenTelemetry trace ID.
- Trace
Query - Trace-by-ID query and bounded search constraints.
- Trace
Stripe - Bounded, single-writer trace state owned by one physical stripe.
- Trace
Summary - Immutable directory entry for one trace’s block fragments and summary.
- Traceql
Engine - Clean-room Rust TraceQL evaluator backed by trace-owner stripes.
- Traceql
Error - TraceQL parse or execution error.
- Traceql
Limits - Bounded TraceQL execution limits.
- Traceql
Metric Exemplar - One trace-linked exemplar emitted by a TraceQL metrics query.
- Traceql
Metric Sample - One TraceQL metrics sample.
- Traceql
Metric Series - One Prometheus-like time series derived directly from matching spans.
- Traceql
Trace - One trace selected by the clean-room TraceQL evaluator.
- Upstream
Offload Config - Configuration for a generic background upstream offloader.
- Upstream
Offload Loop Config - Scheduling and retry bounds for
UpstreamOffloader::run_until. - Upstream
Offloader - Generic background store-and-forward uploader for an embedded local store.
Enums§
- Analytics
Column - One stable column available from at least one analytical relation.
- Analytics
Relation - A stable telemetry relation exposed to ClickHouse.
- Analytics
Scan Order - Exact timestamp order that the analytical storage layer may apply before a bounded log scan is returned to ClickHouse.
- Analytics
Wire Format - Encoding used by the authenticated analytical scan boundary.
- Case
Sensitivity - Case handling for message and metadata text predicates.
- Compression
Codec - Compression format used for one sealed log block.
- Docker
LogStream - Stream value from a Docker
json-filerecord. - Embedded
Eviction Policy - Fate of raw records that leave an embedded node’s recent local window.
- Embedded
Telemetry State - Observable lifecycle state of
EmbeddedTelemetryRuntime. - Histogram
Count - Exact integer or floating-point histogram count.
- Locality
Granularity - Locality class selected for a block.
- LogPredicate
- Exact Boolean predicate evaluated against one normalized log record.
- Metric
Apply Outcome - Result of applying one raw metric point.
- Metric
Ingest Protocol - Ingestion semantics used for same-timestamp sample conflicts.
- Metric
Kind - Metric instrument identity fields that are common to a series.
- Metric
Value - Signal-native metric point payload.
- Native
Opcode - Native operation carried by a frame.
- Native
Query Direction - Sort direction for an indexed native query.
- Native
Status - Status returned in a native response frame.
- Number
Value - Exact scalar number used by gauges, sums, and exemplars.
- Numeric
Comparison - Integer comparison for normalized numeric metadata.
- Promql
Value - Native PromQL result value.
- Query
Order - Ordering applied to matching log records.
- Query
Sort - Stable sort key used by a lookup and its cursor.
- Remote
Write Version - Remote Write protobuf schema selected through the Content-Type parameter.
- Service
State - Runtime lifecycle exposed by the production HTTP and native servers.
- Telemetry
Append Durability - Durability required before ShardTelemetry acknowledges a WAL append.
- Telemetry
Error - Error returned by ShardTelemetry operations.
- Telemetry
Signal - Telemetry signal stored by ShardTelemetry.
- Telemetry
Value - Exact OpenTelemetry attribute value.
- Text
Match Kind - Non-regex text operation used by message and metadata predicates.
- Tier
Artifact Kind - Kind of immutable artifact attached to one block group.
- Trace
Apply Outcome - Result of applying one span to a stripe-local trace head.
Constants§
- ANALYTICS_
SCHEMA_ VERSION - The only pre-release analytical schema and protocol version.
- CLICKHOUSE_
COMPATIBILITY_ TARGET - Pinned ClickHouse release whose evaluator defines ShardTelemetry SQL semantics.
- LOGS_
TOPIC_ ID - Fixed shard-stream topic for log records.
- MAX_
NATIVE_ FRAME_ BYTES - Production maximum for one native request or response payload.
- MAX_
TELEMETRY_ ENVELOPE_ BYTES - Maximum accepted durable envelope, matching the default OTLP request limit.
- METRICS_
TOPIC_ ID - Fixed shard-stream topic for metric points.
- METRIC_
FLAG_ STALE - Internal point flag indicating the Prometheus stale marker.
- NATIVE_
FRAME_ HEADER_ BYTES - Fixed number of bytes in every native protocol frame header.
- PROMETHEUS_
STALE_ NAN_ BITS - Exact Prometheus stale-NaN payload.
- TRACES_
TOPIC_ ID - Fixed shard-stream topic for spans.
Traits§
- Loki
Store - Storage contract used by the Loki-compatible protocol boundary.
- Native
Request Gate - Product-owned admission check evaluated for every native append and query.
- Shard
Stream Durable Sink - Integration point called by the shard-stream worker after durable append.
- Structural
Record View - Read-only normalized record fields consumed by the structural encoder.
- Telemetry
Append Gate - Product-owned admission check evaluated before a durable telemetry append.
- Telemetry
Object Store - Minimal object-store contract needed by the ShardTelemetry tier.
Functions§
- analyze_
message - Scans a message once, calling
on_termfor every Unicode alphanumeric term. - decode_
embedded_ frame_ index - Opens the exact embedded index without reconstructing record bodies or metadata values.
- decode_
log_ envelope - Decodes and verifies the exact log records in one durable v1 envelope.
- decode_
metric_ chunk - Decodes and verifies one signal-native metric chunk.
- decode_
native_ capabilities - Decodes native server capabilities.
- decode_
native_ log_ query_ result - Decodes and fully validates a grouped native log batch.
- decode_
native_ metric_ query - Decodes one bounded native metric query.
- decode_
native_ metric_ query_ result - Decodes native metric query results.
- decode_
native_ query - Decodes and validates an indexed native query.
- decode_
native_ trace_ query - Decodes one bounded native trace query.
- decode_
native_ trace_ query_ result - Decodes native trace query results.
- decode_
signal_ recovery_ state - Decodes the signal-local recovery snapshot from the current pre-release index format.
- decode_
structural_ block - Reconstructs exact record data from one decompressed structural block.
- decode_
structural_ records - Reconstructs only selected record ordinals from one structural block.
- decode_
trace_ block - Decodes and verifies a signal-native trace block.
- encode_
indexed_ structural_ records - Encodes structural data and builds its compressed-domain index in the same template and metadata dictionary pass.
- encode_
metric_ chunk - Encodes one series’ sorted points into a signal-native metric chunk.
- encode_
native_ capabilities - Encodes native server capabilities.
- encode_
native_ log_ query_ result - Encodes records with labels stored once per stream.
- encode_
native_ metric_ query - Encodes one bounded native metric query.
- encode_
native_ metric_ query_ result - Encodes native metric query results.
- encode_
native_ query - Encodes an indexed native query.
- encode_
native_ trace_ query - Encodes one bounded native trace query.
- encode_
native_ trace_ query_ result - Encodes native trace query results.
- encode_
structural_ block - Encodes records into the one pre-release structural block layout.
- encode_
structural_ records - Encodes any zero-copy normalized record view into the current structural block layout.
- encode_
trace_ block - Encodes one partition’s spans into the signal-native columnar trace block.
- fingerprint_
message - Returns a fingerprint without observing search terms.
- is_
native_ telemetry_ batch - Returns true when a native append payload uses the signal-aware v1 batch format.
- loki_
router - Builds the stable Loki 3.7-compatible HTTP route surface.
- loki_
router_ with_ clickhouse - Builds the Loki surface plus the authenticated ClickHouse Arrow scan route.
- mark_
group_ offloaded - Marks all local blocks in a published group as durable payload ranges.
- message_
pattern - Reuses the structural encoder’s lossless token classifier to render a
Loki-compatible pattern with dynamic values replaced by
<_>. - otlp_
http_ router - Builds the OTLP/HTTP router for
/v1/{logs,traces,metrics}. - prepare_
docker_ log_ envelope - Builds a log envelope directly from Docker records without constructing compatibility maps for every record.
- prepare_
docker_ log_ envelope_ with_ context - Builds a Docker log envelope and the process-local index context used by the live owner stripe to avoid decompressing the frame during indexing.
- prepare_
log_ envelope - Builds one durable STEL log envelope from already validated events.
- prepare_
log_ envelope_ with_ context - Builds one durable STEL log envelope plus the process-local index context consumed by a live owner stripe.
- prepare_
loki_ log_ envelope - Converts Loki entries into the single typed log representation and STEL envelope.
- prepare_
metric_ envelope - Builds one durable STEL metric envelope for a single routed partition.
- prepare_
metric_ envelope_ with_ protocol - Builds one durable STEL metric envelope with explicit conflict semantics.
- prepare_
trace_ envelope - Builds one durable STEL trace envelope for a single routed partition.
- prometheus_
router - Builds Prometheus Remote Write and query compatibility routes.
- prometheus_
string_ labels - Returns Prometheus-visible string labels for one canonical series.
- scan_
message_ terms - Scans only case-preserving search terms without calculating compression fingerprints.
- serve_
native - Serves multiplexed native connections until
shutdownresolves. - serve_
otlp_ grpc - Serves OTLP/gRPC logs, traces, and metrics with gzip support.
- single_
tenant_ loki_ api_ router - Builds the authenticated Loki/OTLP compatibility routes for embedding in a host which already owns health, readiness, metrics, and shutdown routes.
- single_
tenant_ loki_ router - Builds the fail-closed single-tenant Loki surface used by the standalone server.
- stage_
signal_ group - Stages one complete trace or metric object-tier group.
- tempo_
router - Builds Tempo v2, search, tag-discovery, and TraceQL routes.
- write_
staged_ payload_ pack - Writes selected staged block payloads as one immutable concatenated pack.
Type Aliases§
- Telemetry
Result - Result returned by ShardTelemetry operations.