Skip to main content

Crate shard_telemetry

Crate shard_telemetry 

Source
Expand description

Signal-native logs, traces, and metrics storage for shard-stream.

A ShardTelemetry owns one single-writer LogStripe per shard-stream physical shard. Call ShardTelemetry::apply_durable from the owning shard-stream worker only after the associated append is durable. This keeps the append log authoritative while allowing term and metadata indexes to be queried through a per-partition indexed watermark.

Structs§

AnalyticsRow
One normalized row shared by the relation-specific Arrow writers.
AnalyticsScanRequest
Bounded storage-level scan requested by ClickHouse.
AttributeFingerprint
Stable 128-bit identity of one exact typed metadata key/value.
BlockCatalog
In-memory directory of sealed blocks owned by one log stripe.
BlockDescriptor
Immutable metadata describing a sealed compression block.
BlockId
Monotonically assigned identifier for a sealed log data block.
BlockQueryIndex
Exact term and metadata postings for one structural block.
CachedObjectRange
One immutable object range backed by a shared verified cache chunk.
CatalogGroupEntry
Bounded catalog entry pointing to one immutable group manifest.
CatalogLease
Read lease for one immutable catalog generation.
CatalogPage
Immutable bounded page of group catalog entries.
CatalogPageRef
Root-level coarse bounds and immutable pointer for one catalog page.
CatalogPointer
Small mutable pointer atomically selecting the authoritative catalog root.
CatalogRoot
Immutable catalog root for one physical-shard logical-partition pair.
CompressionBlockAssignment
Final assignment of one complete sub-block.
CompressionBlockCollator
Fixed-capacity, owner-local block collator.
CompressionBlockScore
Integer score for a complete block or sub-block.
CompressionCohortId
Stable identifier for a compression cohort, such as a service and schema version.
CompressionLocalityConfig
Bounded settings for one stripe-local block collator.
CompressionLocalityRecord
One record presented to block collation.
CompressionLocalityStats
Cumulative diagnostics for one stripe-local collator.
CompressionPlacement
Final owner-local placement decision for one collated block.
CompressionPlacementId
Stable identifier for a final compression placement.
CompressionShardProfile
Read-only diagnostics for one active compression shard.
CompressionTemperature
Integer compression-locality temperature.
CorrelationBlockFilter
Compact immutable filter used to prune cold signal blocks by shared telemetry identity. False positives are possible; false negatives are not.
CorrelationConfig
Hard bounds for one owner stripe’s cross-signal postings.
CorrelationIndex
Preallocated, single-writer correlation postings owned by one stripe.
CorrelationQuery
Cross-signal lookup constraints combined with AND semantics.
CorrelationStats
Snapshot of bounded correlation-index state.
DecodedRemoteWrite
A completely decoded and validated Remote Write request.
DecodedStructuralRecord
A record reconstructed from the single current structural block layout.
DeleteRequest
One durable Loki-compatible logical deletion request.
DictionaryCache
Byte-bounded LRU for immutable compression dictionaries.
DictionaryCatalog
Control-plane catalog for immutable dictionaries and placement assignments.
DictionaryCatalogSnapshot
Immutable control-plane view of published compression dictionaries.
DictionaryId
Stable identifier for an immutable compression dictionary.
DictionaryInsert
Outcome of inserting an immutable compression dictionary into a stripe cache.
DictionaryPublication
Result of atomically publishing an immutable dictionary assignment.
DockerLogRecord
Minimal owned Docker record used by the native ingestion fast path.
DurableLog
One log event accepted after the corresponding shard-stream append is durable.
DurableMetricPoint
One durable raw metric point. Exactly one point consumes one logical offset.
DurableSpan
One durable OTLP span. Exactly one instance consumes one logical offset.
DurableTelemetryConfig
Standalone durable ShardTelemetry configuration.
DurableTelemetryLimits
Bounded memory, journal, and SSD-cache limits for one local durable store.
DurableTelemetryStore
Signal-native store whose acknowledged writes are durable shard-stream appends and whose reads execute on the owning ShardTelemetry stripe workers.
EmbeddedFrameIndex
Lossless compressed-domain candidate index embedded in one structural frame.
EmbeddedTelemetryConfig
Lifecycle configuration for a one-process embedded ShardTelemetry runtime.
EmbeddedTelemetryRuntime
One-owner embedded runtime suitable for a local node telemetry deployment.
ExplicitHistogramValue
Explicit histogram point payload.
ExponentialHistogramBuckets
Positive or negative exponential-histogram buckets.
ExponentialHistogramValue
Exponential histogram point payload.
FastTelemetryConfig
Bounded configuration for a periodic fast-telemetry snapshot.
FastTelemetryExportReport
Counters returned by one embedded snapshot export.
FastTelemetryExporter
Background-driven bridge from a fast-telemetry runtime to local durable storage.
FetchedTelemetryBatch
One source batch recovered from the local WAL for durable upstream offload.
HistogramBucketSpan
One sparse native-histogram bucket span.
IndexReceipt
Result of publishing one durable append into the hot index.
IndexedStructuralBlock
Structural bytes and their already-built compressed-domain index.
LifetimeMetricRollup
Queryable outcome retained locally after raw metric points are removed or archived.
LifetimeRollupReport
Result of one durable local lifetime-rollup checkpoint.
LocalObjectStore
Filesystem implementation of TelemetryObjectStore used for local operation and deterministic testing of S3-style immutable publication.
LogMatch
A visible log record returned by a LogQuery.
LogQuery
Boolean log lookup against one logical partition.
LogRegex
Validated regular expression used by a log predicate.
LogStripe
A lock-free-by-ownership hot index for one shard-stream physical shard.
LokiApiConfig
Configuration for the Loki-compatible HTTP boundary.
LokiApiError
HTTP-boundary failure with a stable response status and safe message.
LokiApiStore
Thread-safe in-memory reference backend used by differential API tests.
LokiEntry
One normalized Loki entry accepted by the compatibility boundary.
MessageFingerprint
Allocation-free message analysis used by routing and term indexing.
MetadataField
A normalized metadata field stored alongside a log message.
MetricExemplar
One metric exemplar nested under a point.
MetricIdentity
Canonical identity of one metric series.
MetricQuery
Native metric selector used by PromQL storage scans and direct APIs.
MetricStripe
Bounded single-writer metric head and immutable raw chunk collection.
NativeCapabilities
Capabilities negotiated before a remote producer is allowed to mark its telemetry integration ready.
NativeClientConfig
Connection and bounded-request settings for ShardTelemetryClient.
NativeClientError
Error returned by the reusable native client.
NativeFrame
One complete native wire frame.
NativeFrameHeader
Validated fixed header for one multiplexed native frame.
NativeLogQueryResult
Decoded stream-grouped native log query result.
NativePartitionAck
Per-partition acknowledgement returned by native protocol v1.
NativePartitionAppend
One routed STEL envelope in a signal-aware native append.
NativeProtocolError
Protocol validation error suitable for a native bad-request response.
NativeQuery
Bounded native indexed-query request.
NativeServerConfig
Runtime limits for the native TCP listener.
NativeTelemetryAppendAck
Atomic native v1 response containing one acknowledgement per partition.
NativeTelemetryBatch
Native protocol v1 append containing one envelope per resulting partition.
ObjectMetadata
Metadata returned for one object-store object.
ObjectStoreStats
Process-local object-tier operation counters shared by all stripe owners.
ObjectTierConfig
Bounded object-tier catalog configuration.
OffloadCheckpoint
Durable state for one upstream-offloaded local WAL partition.
OffloadError
Error returned by the background store-and-forward pipeline.
OffloadLoopReport
Cumulative outcome returned when a background offload loop is stopped.
OffloadReport
Per-round store-and-forward outcome.
OtlpIngestService
Shared OTLP transport implementation backed by the durable telemetry store.
OtlpLogDecoder
Decoder for binary OTLP ExportLogsServiceRequest payloads.
OtlpLogEvent
One normalized OpenTelemetry log event before shard-stream assigns its offset.
OtlpMetricEvent
One fully validated OTLP metric point awaiting its durable offset.
OtlpReceiverConfig
Production OTLP receiver limits and single-tenant identity.
OtlpSinkConfig
Configuration for shard-telemetry’s per-shard native and OTLP index sinks.
OtlpSpanEvent
One fully validated OTLP span awaiting its durable offset.
OtlpTelemetryDecoder
Transport-independent OTLP trace and metric decoder.
PersistentQueryIndex
Immutable query directory for a set of sealed structural blocks.
ProductionMetricsSnapshot
Monotonic protocol counters exposed by the Prometheus endpoint.
ProductionRuntime
Shared single-tenant admission and lifecycle state for every protocol.
PrometheusApiConfig
Single-tenant Prometheus compatibility limits.
PrometheusService
Shared Prometheus protocol service backed by signal-native metric stripes.
PromqlEngine
Rust PromQL evaluator backed by ShardTelemetry metric stripes.
PromqlError
Parse or evaluation error returned through the Prometheus API envelope.
PromqlLimits
Limits for one native PromQL evaluation.
PromqlSample
One Prometheus float sample with its complete label set.
PromqlSeries
One matrix series returned by a range query.
QueryBlockMetadata
Offset and timestamp bounds for one independently compressed block.
QueryCursor
Exclusive continuation point for deterministic lookup pagination.
QueryHit
Exact location of a candidate record inside a sealed block.
RealtimeDictionaryConfig
Bounded policy for continuously training immutable Zstandard dictionaries.
RealtimeDictionaryObserver
Cloneable, non-blocking block-level submission handle for log stripes.
RealtimeDictionaryStats
Snapshot of online dictionary-learning activity.
RealtimeDictionaryTrainer
Background real-time dictionary learner and immutable publication service.
RemoteWriteDecoder
Lossless decoder for Remote Write v1 and v2 protobuf messages.
RemoteWriteStats
Counts reported in the mandatory Remote Write 2.0 response headers.
ResourceContext
Exact resource context shared by records from one OTLP resource group.
ResourceContextId
Stable 128-bit identity of an exact resource context.
RetentionReport
Result of one batch-aligned physical retention pass.
RetiredObject
One exact object whose previous catalog generation relinquished ownership.
S3ObjectStore
Rust-native S3 adapter with conditional publication and streaming uploads.
S3ObjectStoreConfig
Production S3 and S3-compatible object-store configuration.
ScopeContext
Exact instrumentation scope context shared by records from one OTLP scope.
ScopeContextId
Stable 128-bit identity of an exact instrumentation scope context.
SeriesAccumulatorCheckpoint
Checkpointed delta-to-cumulative state for PromQL views.
SeriesFingerprint
Stable 128-bit identity of a metric series.
ServiceLifecycle
Shared lifecycle controller for HTTP, native TCP, and process shutdown.
ShardTelemetry
Container for independently owned ShardTelemetry log stripes.
ShardTelemetryClient
Persistent, bounded native-v1 client for ShardTelemetry.
ShardTelemetryConfig
Complete single-node ShardTelemetry configuration.
SharedTelemetryObjectStore
Cloneable type-erased object-store handle used by production stripe owners.
SignalConfig
Bounded configuration shared by a single telemetry signal.
SignalTierPayload
One already encoded trace block or metric chunk awaiting immutable publication.
SingleTenantConfig
Fail-closed limits and credentials for one production tenant.
SinkObjectTierConfig
Immutable object-tier and bounded SSD-cache settings shared by sink stripes.
SpanEvent
One nested span event. It does not consume a shard-stream offset.
SpanId
A validated 64-bit OpenTelemetry span ID.
SpanLink
One nested link to another span. It does not consume a shard-stream offset.
SpanStatus
Final OpenTelemetry span status.
SsdCacheConfig
Configuration for one byte-bounded SSD object-range cache.
SsdCacheStats
Runtime diagnostics for one recoverable SSD object cache.
SsdObjectCache
Recoverable, integrity-checked SSD cache for immutable object ranges.
StoreHealth
Health snapshot supplied by a Loki storage backend.
StoreMetrics
Storage counters rendered with protocol counters by /metrics.
StripeConfig
Resource limits for one shard-aligned log stripe.
StructuralLogMetadataRef
Borrowed exact OTLP metadata exposed to the structural encoder.
SummaryQuantileValue
One legacy summary quantile.
SummaryValue
Legacy summary point payload.
TelemetryAttribute
One exact OTLP key/value pair.
TelemetryEntityRef
An OTLP Resource entity reference.
TelemetryEnvelope
One checksummed durable telemetry append payload.
TelemetryHostAttachment
Externally owned shard-stream engine and signal-service attachment.
TelemetryObjectTier
Partition-scoped immutable object tier with a conditionally published root.
TelemetryRecordRef
Stable address of one log record in shard-stream’s ordered append log.
TelemetryRouter
Deterministic signal-aware logical partition router.
TelemetryService
Cloneable read service for the stripes owned by durable sink workers.
TelemetrySinkFactory
Builds one owner-only log index worker for every shard-stream physical shard.
TempoApiConfig
Single-tenant Tempo compatibility limits.
TempoService
Shared Tempo-compatible query service.
TextMatcher
Reusable literal text matcher.
TierArtifact
Immutable object metadata for one published group artifact.
TierArtifactSource
Local source file to publish as one immutable group artifact.
TierBlockEntry
Durable block metadata and payload extent inside a block-group pack.
TierCheckpoint
Durable sink watermark covered by an immutable object-tier group.
TierGroupManifest
Immutable manifest for one independently queryable block group.
TierGroupSource
Complete local input needed to publish one block group.
TierQueryRange
Coarse inclusive bounds used to prune catalog pages and groups.
TierRetentionReport
Result of one bounded exact-key object-tier retention transaction.
TraceDirectory
In-memory view of the immutable trace directory.
TraceId
A validated 128-bit OpenTelemetry trace ID.
TraceQuery
Trace-by-ID query and bounded search constraints.
TraceStripe
Bounded, single-writer trace state owned by one physical stripe.
TraceSummary
Immutable directory entry for one trace’s block fragments and summary.
TraceqlEngine
Clean-room Rust TraceQL evaluator backed by trace-owner stripes.
TraceqlError
TraceQL parse or execution error.
TraceqlLimits
Bounded TraceQL execution limits.
TraceqlMetricExemplar
One trace-linked exemplar emitted by a TraceQL metrics query.
TraceqlMetricSample
One TraceQL metrics sample.
TraceqlMetricSeries
One Prometheus-like time series derived directly from matching spans.
TraceqlTrace
One trace selected by the clean-room TraceQL evaluator.
UpstreamOffloadConfig
Configuration for a generic background upstream offloader.
UpstreamOffloadLoopConfig
Scheduling and retry bounds for UpstreamOffloader::run_until.
UpstreamOffloader
Generic background store-and-forward uploader for an embedded local store.

Enums§

AnalyticsColumn
One stable column available from at least one analytical relation.
AnalyticsRelation
A stable telemetry relation exposed to ClickHouse.
AnalyticsScanOrder
Exact timestamp order that the analytical storage layer may apply before a bounded log scan is returned to ClickHouse.
AnalyticsWireFormat
Encoding used by the authenticated analytical scan boundary.
CaseSensitivity
Case handling for message and metadata text predicates.
CompressionCodec
Compression format used for one sealed log block.
DockerLogStream
Stream value from a Docker json-file record.
EmbeddedEvictionPolicy
Fate of raw records that leave an embedded node’s recent local window.
EmbeddedTelemetryState
Observable lifecycle state of EmbeddedTelemetryRuntime.
HistogramCount
Exact integer or floating-point histogram count.
LocalityGranularity
Locality class selected for a block.
LogPredicate
Exact Boolean predicate evaluated against one normalized log record.
MetricApplyOutcome
Result of applying one raw metric point.
MetricIngestProtocol
Ingestion semantics used for same-timestamp sample conflicts.
MetricKind
Metric instrument identity fields that are common to a series.
MetricValue
Signal-native metric point payload.
NativeOpcode
Native operation carried by a frame.
NativeQueryDirection
Sort direction for an indexed native query.
NativeStatus
Status returned in a native response frame.
NumberValue
Exact scalar number used by gauges, sums, and exemplars.
NumericComparison
Integer comparison for normalized numeric metadata.
PromqlValue
Native PromQL result value.
QueryOrder
Ordering applied to matching log records.
QuerySort
Stable sort key used by a lookup and its cursor.
RemoteWriteVersion
Remote Write protobuf schema selected through the Content-Type parameter.
ServiceState
Runtime lifecycle exposed by the production HTTP and native servers.
TelemetryAppendDurability
Durability required before ShardTelemetry acknowledges a WAL append.
TelemetryError
Error returned by ShardTelemetry operations.
TelemetrySignal
Telemetry signal stored by ShardTelemetry.
TelemetryValue
Exact OpenTelemetry attribute value.
TextMatchKind
Non-regex text operation used by message and metadata predicates.
TierArtifactKind
Kind of immutable artifact attached to one block group.
TraceApplyOutcome
Result of applying one span to a stripe-local trace head.

Constants§

ANALYTICS_SCHEMA_VERSION
The only pre-release analytical schema and protocol version.
CLICKHOUSE_COMPATIBILITY_TARGET
Pinned ClickHouse release whose evaluator defines ShardTelemetry SQL semantics.
LOGS_TOPIC_ID
Fixed shard-stream topic for log records.
MAX_NATIVE_FRAME_BYTES
Production maximum for one native request or response payload.
MAX_TELEMETRY_ENVELOPE_BYTES
Maximum accepted durable envelope, matching the default OTLP request limit.
METRICS_TOPIC_ID
Fixed shard-stream topic for metric points.
METRIC_FLAG_STALE
Internal point flag indicating the Prometheus stale marker.
NATIVE_FRAME_HEADER_BYTES
Fixed number of bytes in every native protocol frame header.
PROMETHEUS_STALE_NAN_BITS
Exact Prometheus stale-NaN payload.
TRACES_TOPIC_ID
Fixed shard-stream topic for spans.

Traits§

LokiStore
Storage contract used by the Loki-compatible protocol boundary.
NativeRequestGate
Product-owned admission check evaluated for every native append and query.
ShardStreamDurableSink
Integration point called by the shard-stream worker after durable append.
StructuralRecordView
Read-only normalized record fields consumed by the structural encoder.
TelemetryAppendGate
Product-owned admission check evaluated before a durable telemetry append.
TelemetryObjectStore
Minimal object-store contract needed by the ShardTelemetry tier.

Functions§

analyze_message
Scans a message once, calling on_term for every Unicode alphanumeric term.
decode_embedded_frame_index
Opens the exact embedded index without reconstructing record bodies or metadata values.
decode_log_envelope
Decodes and verifies the exact log records in one durable v1 envelope.
decode_metric_chunk
Decodes and verifies one signal-native metric chunk.
decode_native_capabilities
Decodes native server capabilities.
decode_native_log_query_result
Decodes and fully validates a grouped native log batch.
decode_native_metric_query
Decodes one bounded native metric query.
decode_native_metric_query_result
Decodes native metric query results.
decode_native_query
Decodes and validates an indexed native query.
decode_native_trace_query
Decodes one bounded native trace query.
decode_native_trace_query_result
Decodes native trace query results.
decode_signal_recovery_state
Decodes the signal-local recovery snapshot from the current pre-release index format.
decode_structural_block
Reconstructs exact record data from one decompressed structural block.
decode_structural_records
Reconstructs only selected record ordinals from one structural block.
decode_trace_block
Decodes and verifies a signal-native trace block.
encode_indexed_structural_records
Encodes structural data and builds its compressed-domain index in the same template and metadata dictionary pass.
encode_metric_chunk
Encodes one series’ sorted points into a signal-native metric chunk.
encode_native_capabilities
Encodes native server capabilities.
encode_native_log_query_result
Encodes records with labels stored once per stream.
encode_native_metric_query
Encodes one bounded native metric query.
encode_native_metric_query_result
Encodes native metric query results.
encode_native_query
Encodes an indexed native query.
encode_native_trace_query
Encodes one bounded native trace query.
encode_native_trace_query_result
Encodes native trace query results.
encode_structural_block
Encodes records into the one pre-release structural block layout.
encode_structural_records
Encodes any zero-copy normalized record view into the current structural block layout.
encode_trace_block
Encodes one partition’s spans into the signal-native columnar trace block.
fingerprint_message
Returns a fingerprint without observing search terms.
is_native_telemetry_batch
Returns true when a native append payload uses the signal-aware v1 batch format.
loki_router
Builds the stable Loki 3.7-compatible HTTP route surface.
loki_router_with_clickhouse
Builds the Loki surface plus the authenticated ClickHouse Arrow scan route.
mark_group_offloaded
Marks all local blocks in a published group as durable payload ranges.
message_pattern
Reuses the structural encoder’s lossless token classifier to render a Loki-compatible pattern with dynamic values replaced by <_>.
otlp_http_router
Builds the OTLP/HTTP router for /v1/{logs,traces,metrics}.
prepare_docker_log_envelope
Builds a log envelope directly from Docker records without constructing compatibility maps for every record.
prepare_docker_log_envelope_with_context
Builds a Docker log envelope and the process-local index context used by the live owner stripe to avoid decompressing the frame during indexing.
prepare_log_envelope
Builds one durable STEL log envelope from already validated events.
prepare_log_envelope_with_context
Builds one durable STEL log envelope plus the process-local index context consumed by a live owner stripe.
prepare_loki_log_envelope
Converts Loki entries into the single typed log representation and STEL envelope.
prepare_metric_envelope
Builds one durable STEL metric envelope for a single routed partition.
prepare_metric_envelope_with_protocol
Builds one durable STEL metric envelope with explicit conflict semantics.
prepare_trace_envelope
Builds one durable STEL trace envelope for a single routed partition.
prometheus_router
Builds Prometheus Remote Write and query compatibility routes.
prometheus_string_labels
Returns Prometheus-visible string labels for one canonical series.
scan_message_terms
Scans only case-preserving search terms without calculating compression fingerprints.
serve_native
Serves multiplexed native connections until shutdown resolves.
serve_otlp_grpc
Serves OTLP/gRPC logs, traces, and metrics with gzip support.
single_tenant_loki_api_router
Builds the authenticated Loki/OTLP compatibility routes for embedding in a host which already owns health, readiness, metrics, and shutdown routes.
single_tenant_loki_router
Builds the fail-closed single-tenant Loki surface used by the standalone server.
stage_signal_group
Stages one complete trace or metric object-tier group.
tempo_router
Builds Tempo v2, search, tag-discovery, and TraceQL routes.
write_staged_payload_pack
Writes selected staged block payloads as one immutable concatenated pack.

Type Aliases§

TelemetryResult
Result returned by ShardTelemetry operations.