monitord
monitord ... know how happy your systemd is! 😊
Requirements
- Linux with systemd (monitord uses D-Bus and procfs APIs that are Linux-specific)
- systemd-networkd installed (for networkd metrics; the collector can be disabled in config)
- PID 1 stats require procfs (
/proc) — available on all standard Linux systems - D-Bus system bus accessible (default:
unix:path=/run/dbus/system_bus_socket) - Varlink metrics require systemd v260+ (optional; falls back to D-Bus automatically)
What does monitord monitor?
monitord collects systemd health metrics via D-Bus (and optionally Varlink) and outputs them as JSON. It provides visibility into:
- Unit counts — totals by type (service, mount, socket, timer, etc.) and state (active, failed, inactive, loaded, masked)
- Per-service stats — CPU usage, memory, I/O, restart count, task count, watchdog status, and state timestamps for specific services
- Unit state tracking — active state, load state, and health for individual units (with allowlist/blocklist filtering)
- systemd-networkd — per-interface operational, carrier, admin, and address states
- PID 1 health — CPU time, memory usage, file descriptor count, and task count for systemd (PID 1) via procfs
- Timers — trigger times, accuracy, delays, and associated service state for systemd timers
- Boot blame — the N slowest units at boot, similar to
systemd-analyze blame - D-Bus daemon stats — connection counts, match rules, and per-peer/per-cgroup/per-user breakdowns (dbus-broker and dbus-daemon)
- Containers / machines — recursively collects the same metrics from systemd-nspawn containers and VMs via
systemd-machined - Unit verification — runs
systemd-analyze verifyand reports failing unit counts by type
Run Modes
We offer the following run modes:
- systemd-timer (legacy cron would work too)
- Refer to monitord.timer and monitord.service unit files
- Ensure no
daemon:mode options are set inmonitord.conf
- daemon mode
- Enable daemon mode in configuration file
- Stats will be written to stdout every
daemon_stats_refresh_secs
Open to more formats / run methods ... Open an issue to discuss. Depends on the dependencies basically.
monitord is a config driven binary. We plan to keep CLI arguments to a minimum.
INFO level logging is enabled to stderr by default. Use -l LEVEL to increase or decrease logging.
Quick Start
-
Install monitord:
-
Create a minimal config at
/etc/monitord.conf:[monitord] output_format = json-pretty [units] enabled = true [pid1] enabled = true -
Run it:
This will collect unit counts and PID 1 stats, then print JSON to stdout and exit. Enable additional collectors in the config as needed (see Configuration below).
Install
Pre-built binaries
Download pre-built binaries from GitHub Releases:
monitord-linux-amd64— x86_64monitord-linux-aarch64— ARM64
# Example: download and install the latest release (x86_64)
From crates.io
Install via cargo or use as a dependency in your Cargo.toml.
cargo install monitord- Create (copy from repo) a
monitord.conf- Defaults to looking for it at /etc/monitord.conf
monitord --help- Also support
MONITORD_CONFIGenv var to set config path
- Also support
crl-linux:monitord cooper$ monitord --help
monitord: Know how happy your systemd is! 😊
Usage: monitord [OPTIONS]
Options:
-c, --config <CONFIG>
Location of your monitord config
[default: /etc/monitord.conf]
-l, --log-level <LOG_LEVEL>
Adjust the console log-level
[default: Info]
[possible values: error, warn, info, debug, trace]
-h, --help
Print help (see a summary with '-h')
-V, --version
Print version
Config
monitord can have the different components monitored. To enable / disabled set the following in our monitord.conf. This file is ini format to match systemd unit files.
# Pure ini - no yes/no for bools
[monitord]
# Set a custom dbus address to connect to
# OPTIONAL: If not set, we default to the Unix socket below
dbus_address = unix:path=/run/dbus/system_bus_socket
# Timeout in seconds for dbus connection/collections
# OPTIONAL: default is 30 seconds
dbus_timeout = 30
# Run as a daemon or 1 time
daemon = false
# Time to refresh systemd stats in seconds
# Daemon mode only
daemon_stats_refresh_secs = 60
# Prefix flat-json key with this value
# The value automatically gets a '.' appended (so don't put here)
key_prefix = monitord
# cron/systemd timer output format
# Supported: json, json-flat, json-pretty
output_format = json
# Grab as much stats from DBus GetStats call
# we can from running dbus daemon
# More tested on dbus-broker daemon
[dbus]
# Summary counters - both dbus-broker + dbus-daemon
enabled = false
# dbus.user.* metrics: user stats as reported by dbus-broker
user_stats = false
# dbus.oeer.* metrics: peer stats as reported by dbus-broker
peer_stats = false
# dbus.cgroup.* stats is an aggregation of peer_stats by cgroup
# by dbus-broker
cgroup_stats = false
# Grab networkd stats from files + networkctl
[networkd]
enabled = true
link_state_dir = /run/systemd/netif/links
# Enable grabbing PID 1 stats via procfs
[pid1]
enabled = true
# Services to grab extra stats for
# .service is important as that's what DBus returns from `list_units`
[services]
foo.service
[timers]
enabled = true
[timers.allowlist]
foo.timer
[timers.blocklist]
bar.timer
# Grab unit status counts via dbus
[units]
enabled = true
state_stats = true
# Filter what services you want collect state stats for
# If both lists are configured blocklist is preferred
# If neither exist all units state will generate counters
[units.state_stats.allowlist]
foo.service
[units.state_stats.blocklist]
bar.service
# machines config
[machines]
enabled = true
# Same rules apply as state_stats lists above
[machines.allowlist]
foo
[machines.blocklist]
bar
# Boot blame metrics - shows the N slowest units at boot
# Similar to `systemd-analyze blame`
# Disabled by default
[boot]
enabled = false
# Cache boot blame stats in /run/monitord/<boot_id>.boot_blame.bin
# Enabled by default; set false to force recalculation every run
cache_enabled = true
# Number of slowest units to report
num_slowest_units = 5
# Optional: only include specific units in boot blame (if empty, all units are checked)
# Same rules apply as state_stats lists above
[boot.allowlist]
# slow-startup.service
# Optional: exclude specific units from boot blame
[boot.blocklist]
# noisy-but-expected.service
# Unit verification using systemd-analyze verify
# Disabled by default as it can be slow on large systems
[verify]
enabled = false
# Optional: only verify specific units (if empty, all units are checked)
[verify.allowlist]
# example.service
# example.timer
# Optional: skip verification for specific units
[verify.blocklist]
# noisy.service
# broken.timer
When using the provided monitord.service, systemd creates /run/monitord via
RuntimeDirectory=monitord and assigns ownership to the configured service User/Group.
If you run monitord another way, ensure /run/monitord exists and is writable by the
monitord process user so boot cache files can be created.
Machines support
From version >=0.11 monitord supports obtaining the same set of key from
systemd 'machines' (i.e. machinectl --list).
The keys are the same format as below in json_flat output but are prefixed with
the machine keyword and machine name. For example:
# $KEY_PREFIX.machine.$MACHINE_NAME
Output Formats
json
Normal serde_json non pretty JSON. All on one line. Most compact format.
json-flat
Move all key value pairs to the top level and . notate components + sub values. Is semi pretty too + custom. All unittested ...
stat_collection_run_time_ms is emitted in milliseconds (with _ms suffix) to follow
Prometheus metric naming conventions for duration units, which keeps unit semantics
clear and consistent when these keys are transformed into Prometheus metric names.
json-pretty
Normal serde_json pretty representations of each components structs.
Per-collector timing metrics
monitord records the wall time each collector future spends inside a single
stat_collector cycle and exposes the result on MonitordStats::collector_timings,
plus an inner phase breakdown for the units collector
(SystemdUnitStats::collection_timings).
| Field | Meaning |
|---|---|
collector_timings.<name>.start_offset_ms |
ms from the top of the cycle until the spawned future was first polled. Should be sub-ms when collectors are running in parallel; a non-trivial value means the spawn loop or runtime is delaying first poll. |
collector_timings.<name>.elapsed_ms |
ms from first poll to completion for that collector. |
collector_timings.<name>.success |
1 if the collector returned Ok, 0 otherwise. |
collection_timings.list_units_ms |
ms for the systemd ListUnits D-Bus call (one batched call). |
collection_timings.per_unit_loop_ms |
ms spent walking each listed unit, including any per-unit D-Bus calls (timer/state/service). |
collection_timings.timer_dbus_fetches |
Count of timer D-Bus property fetches this run. |
collection_timings.state_dbus_fetches |
Count of unit-state D-Bus fetches (only when state_stats_time_in_state is enabled). |
collection_timings.service_dbus_fetches |
Count of per-service D-Bus property fetches. |
Comparing sum(collector_timings.*.elapsed_ms) against
stat_collection_run_time_ms gives an effective parallelism ratio
(sum / wall ≈ N means N-way parallelism, ≈ 1 means effectively serial).
The per-collector lines are also emitted to logs at debug! level. The end-of-cycle
"stat collection run took {}ms" summary stays at info!.
Varlink-vs-D-Bus parity
collection_timings is populated identically by the D-Bus path
(units::parse_unit_state) and the varlink path
(varlink_units::parse_metrics). In the varlink case, list_units_ms is the
bulk varlink List call on io.systemd.Manager and per_unit_loop_ms is the
local parse loop; the *_dbus_fetches counters stay at zero, which is itself a
useful signal that the varlink path is not paying per-unit D-Bus cost. This
makes varlink.enabled = true vs false directly comparable on the same host.
Convention for new collectors moved to varlink: when porting a collector
from D-Bus to varlink, add the equivalent inner timings so the two
implementations remain comparable. The minimum is wall time of the bulk
fetch (analogous to list_units_ms) and the local parse loop (analogous to
per_unit_loop_ms), recorded onto a struct nested inside the collector's
public stats type. Single-shot varlink calls (e.g. networkd Describe) do
not need an inner split — the outer collector_timings.<name>.elapsed_ms
already covers them.
Metric Value Reference
Many metrics are serialized as integers. Here are the enum mappings:
system-state
| Value | State |
|---|---|
| 0 | unknown |
| 1 | initializing |
| 2 | starting |
| 3 | running |
| 4 | degraded |
| 5 | maintenance |
| 6 | stopping |
| 7 | offline |
active_state (unit_states.*.active_state)
| Value | State |
|---|---|
| 0 | unknown |
| 1 | active |
| 2 | reloading |
| 3 | inactive |
| 4 | failed |
| 5 | activating |
| 6 | deactivating |
loaded_state (unit_states.*.loaded_state)
| Value | State |
|---|---|
| 0 | unknown |
| 1 | loaded |
| 2 | error |
| 3 | masked |
| 4 | not-found |
networkd address_state / ipv4_address_state / ipv6_address_state
| Value | State |
|---|---|
| 0 | unknown |
| 1 | off |
| 2 | degraded |
| 3 | routable |
networkd admin_state
| Value | State |
|---|---|
| 0 | unknown |
| 1 | pending |
| 2 | failed |
| 3 | configuring |
| 4 | configured |
| 5 | unmanaged |
| 6 | linger |
networkd carrier_state
| Value | State |
|---|---|
| 0 | unknown |
| 1 | off |
| 2 | no-carrier |
| 3 | dormant |
| 4 | degraded-carrier |
| 5 | carrier |
| 6 | enslaved |
networkd oper_state
| Value | State |
|---|---|
| 0 | unknown |
| 1 | missing |
| 2 | off |
| 3 | no-carrier |
| 4 | dormant |
| 5 | degraded-carrier |
| 6 | carrier |
| 7 | degraded |
| 8 | enslaved |
| 9 | routable |
dbus stats
You're going to need to be root or allow permissiong to pull dbus stats.
For dbus-broker here is example config allow a user monitord to query
getStats
[cooper@l33t ~]# cat /etc/dbus-1/system.d/allow_monitord_stats.conf
Development
To do test runs (requires systemd and systemd-networkd installed)
-
Pending what you have enabled in your config ...
-
cargo run -- -c monitord.conf -l debug
Ensure the following pass before submitting a PR (CI checks):
cargo testcargo clippycargo fmt
Releasing a new version
- Increment the version in
Cargo.toml - Run
./build_docs.shto regenerate docs - Commit with message:
Move to version X.Y.Z for release + update docs - If you have commit bit, push directly to main. Otherwise, push a branch and open a PR.
- Cut a GitHub release:
gh release create X.Y.Z --title "X.Y.Z" --generate-notes
Generate codegen APIs
cargo install zbus_xmlgenzbus-xmlgen system org.freedesktop.systemd1 /org/freedesktop/systemd1/unit/chronyd_2eservice
Then add the following macros to tell clippy to go away:
Non Linux development
Sometimes I develop from my Mac OS X laptop. So I thought I'd document and add the way I build a Fedora Rawhide container and mount the local repo to /repo in the container to run monitord and test.
- Build the image (w/git, rust tools and systemd)
docker build -t monitord-dev .
- Start via systemd and mount the monitord repo to /repo
docker run --rm --name monitord-dev -it --privileged --tmpfs /run --tmpfs /tmp -v $(pwd):/repo monitord-dev /sbin/init--rmis optional but will remove the container when stopped
You can now log into the container to build + run tests and run the binary now against systemd.
docker exec -it monitord-dev bashcd /repo ; cargo run -- -c monitord- networkd etc. are not running my default but can be started ...
systemctl start systemd-networkd- No interfaces will be managed tho by default in the container ...
Troubleshooting
"Connection refused" or D-Bus connection errors
Ensure the system D-Bus daemon is running and the socket exists at /run/dbus/system_bus_socket. If using a custom address, set dbus_address in [monitord] config. Increase dbus_timeout if running on slow systems.
Empty or missing networkd metrics
systemd-networkd must be installed and running (systemctl start systemd-networkd). If networkd is not in use on your system, disable the collector with enabled = false in [networkd].
Permission denied for D-Bus stats
The [dbus] collector requires permission to call org.freedesktop.DBus.Debug.Stats.GetStats. Either run monitord as root or add a D-Bus policy file — see the dbus stats section.
PID 1 stats unavailable
PID 1 stats require Linux with procfs mounted at /proc. This collector is compiled out on non-Linux targets. If /proc is not available (some container runtimes), disable with enabled = false in [pid1].
Collector errors don't crash monitord
When an individual collector fails (e.g., networkd not running, D-Bus timeout), monitord logs a warning and continues with the remaining collectors. Check stderr output or increase the log level (-l debug) to see which collectors had issues.
Large u64 values (18446744073709551615) in output
These represent u64::MAX and mean "not available" or "not tracked" for that metric. This is how systemd reports fields that are unsupported or not configured for the unit (e.g., memory_available when MemoryMax= is not set).
Library API
monitord can be used as a Rust library. See the full API documentation at monitord.xyz.
DBus
All monitord's dbus is done via async (tokio) zbus crate.
systemd Dbus APIs are in use in the following modules:
- machines
ManagerProxy::list_machines()- Can do most other calls then on the machine's systemd/dbus
- networkd
ManagerProxy::list_links()- Interface state files at
/run/systemd/netif/linksare used by default; the varlinkio.systemd.Network.DescribeAPI can be enabled instead (see below)
- system
ManagerProxy::get_version()ManagerProxy::system_state()
- timer
TimerProxy::unit()- Find service unit of timerManagerProxy::get_unit()UnitProxy::state_change_timestamp()UnitProxy::state_change_timestamp_monotonic()
- units
ManagerProxy::list_units()- Main counting of unit statsServiceProxy::cpuusage_nsec()ServiceProxy::ioread_bytes()ServiceProxy::ioread_operations()ServiceProxy::memory_current()ServiceProxy::memory_available()ServiceProxy::nrestarts()ServiceProxy::get_processes()ServiceProxy::restart_usec()ServiceProxy::status_errno()ServiceProxy::tasks_current()ServiceProxy::timeout_clean_usec()ServiceProxy::watchdog_usec()UnitProxy::active_enter_timestampUnitProxy::active_exit_timestampUnitProxy::inactive_exit_timestamp()UnitProxy::state_change_timestamp()- Used for raw stat + time_in_state
Some of these modules can be disabled via configuration. Due to this, monitord might not always be running / calling all these DBus calls per run.
Varlink
monitord supports collecting unit statistics via systemd's Varlink metrics API,
available in systemd v260+. When enabled, monitord connects to the io.systemd.Metrics interface
at /run/systemd/report/io.systemd.Manager to collect unit counts, active/load states, and restart counts.
Enabling Varlink
Set enabled = true in the [varlink] section of monitord.conf:
[varlink]
enabled = true
When varlink is enabled, monitord will attempt to collect stats via the varlink APIs first, automatically falling back to D-Bus or file-based collection when a varlink socket is unavailable (e.g., older systemd versions).
Metrics collected via Varlink
Units (io.systemd.Metrics — systemd v260+):
- Unit counts by type (service, mount, socket, target, device, automount, timer, path, slice, scope)
- Unit counts by state (active, failed, inactive)
- Per-unit active state and load state (with allowlist/blocklist filtering)
- Per-unit health status (computed from active + load state)
- Per-service restart counts (
nrestarts) - Falls back to D-Bus collection if the socket is unavailable
Networkd interfaces (io.systemd.Network.Describe — systemd v257+):
- Per-interface operational, carrier, admin, and address states
- Falls back to parsing
/run/systemd/netif/linksstate files if the socket is unavailable
Containers
For systemd-nspawn containers, monitord connects to the container's varlink socket via
/proc/<leader_pid>/root/run/systemd/report/io.systemd.Manager, similar to how D-Bus uses
the container-scoped bus socket. Networkd stats use
/proc/<leader_pid>/root/run/systemd/netif/io.systemd.Network, with the same file-based fallback.
varlink 101
varlink might one day replace our DBUS usage. Here are some notes on how to work with systemd varlink
as there isn't really documentation outside man pages.
Checking interfaces
- varlinkctl is your friend - https://man7.org/linux/man-pages/man1/varlinkctl.1.html
Here is an example with networkd's interfaces:
varlinkctl info unix:/run/systemd/netif/io.systemd.Network
varlinkctl introspect unix:/run/systemd/netif/io.systemd.Network io.systemd.Network
cooper@au:~$ varlinkctl call unix:/run/systemd/netif/io.systemd.Network io.systemd.Network.GetStates '{}' -j | jq
{
"AddressState": "routable",
"IPv4AddressState": "routable",
"IPv6AddressState": "routable",
"CarrierState": "carrier",
"OnlineState": "online",
"OperationalState": "routable"
}