Crate micromegas

Source
Expand description

Micromegas is a unified and scalable observability stack. It can help you collect and query logs, metrics and traces.

§Very high level architecture

┌─────────────────┐       
│ rust application│──────▶
└─────────────────┘       ┌─────────┐     ┌───────┐     ┌─────────┐     ┌──────────┐
                          │ingestion│────▶│pg & S3│◀────│analytics│◀────│python API│
┌─────────────────┐       └─────────┘     └───────┘     └─────────┘     └──────────┘
│ unreal engine   │──────▶
└─────────────────┘

§Rust Instrumentation

For rust applications, use micromegas::tracing for minimal overhead. Interoperability with tokio tracing’s logs is also enabled by default.

§Unreal instrumentation

MicromegasTracing should be added to Unreal’s Core module and MicromegasTelemetrySink can be added to a game or to a high level plugin. See https://github.com/madesroches/micromegas/tree/main/unreal for implementation. It has been tested in editor, client and server builds on multiple platforms.

§Telemetry ingestion server

https://github.com/madesroches/micromegas/blob/main/rust/telemetry-ingestion-srv/src/main.rs

§FlightSQL server

https://github.com/madesroches/micromegas/blob/main/rust/flight-sql-srv/src/flight_sql_srv.rs

§Lakehouse daemon

https://github.com/madesroches/micromegas/blob/main/rust/telemetry-admin-cli/src/telemetry_admin.rs (with crond argument)

§Python API

https://pypi.org/project/micromegas/

§Local developer configuration

For testing purposes, you can run the entire stack on your local workstation.

§Environment variables

  • MICROMEGAS_DB_USERNAME and MICROMEGAS_DB_PASSWD: used by the database configuration script
  • export MICROMEGAS_TELEMETRY_URL=http://localhost:9000
  • export MICROMEGAS_SQL_CONNECTION_STRING=postgres://{uname}:{passwd}@localhost:5432
  • export MICROMEGAS_OBJECT_STORE_URI=file:///some/local/path
  1. Clone the github repository
> git clone https://github.com/madesroches/micromegas.git
  1. Start a local instance of postgresql (requires docker and python)
> cd micromegas/local_test_env/db
> ./run.py
  1. In a new shell, start the ingestion server
> cd micromegas/rust
> cargo run -p telemetry-ingestion-srv -- --listen-endpoint-http 127.0.0.1:9000
  1. In a new shell, start the flightsql server
> cd micromegas/rust
> cargo run -p flight-sql-srv -- --disable-auth
  1. In a new shell, start the daemon
> cd micromegas/rust
> cargo run -p telemetry-admin -- crond
  1. In a python interpreter, query the analytics service
# local connection test
import datetime
import micromegas
client = micromegas.connect() #connects to localhost by default
now = datetime.datetime.now(datetime.timezone.utc)
begin = now - datetime.timedelta(days=1)
end = now
sql = """
SELECT *
FROM log_entries
ORDER BY time DESC
LIMIT 10
;"""
df = client.query(sql, begin, end)
df #dataframe containing the result of the query

Re-exports§

pub use arrow_flight;
pub use axum;
pub use chrono;
pub use datafusion;
pub use object_store;
pub use prost;
pub use sqlx;
pub use tonic;
pub use uuid;

Modules§

analytics
makes the telemetry data lake accessible and useful
axum_utils
axum-utils : observability middleware
client
rust analytics client
ingestion
records telemetry in data lake
key_ring
named keys for authentication
log_uri_service
log uris of http requests
perfetto
perfetto protobufs
servers
Embedable ingestion, analytics and maintenance services. The user is expected to provide their own authentication.
telemetry
telemetry protocol
telemetry_sink
publication of the recorded events using http
tonic_auth_interceptor
authentication for the gRPC stack
tracing
low level tracing - has minimal dependencies