zoter 0.2.0

Restart systemd services without dropping their clients
Documentation

Restart systemd services without dropping their clients.

A service that stops hands its state and the connections of its clients over to its next instance, through the file descriptor store that systemd keeps for the service across its restarts. The clients notice nothing but a pause: the next instance serves them on the same connections, from where the previous one stopped, while those that connect meanwhile wait in the socket of socket activation, which systemd keeps open.

zoter does what a service needs for that, except what depends on the service's protocol, such as D-Bus or Varlink:

  • Passed takes the file descriptors that systemd passes a service: the sockets of socket activation, and what the previous instance stored.
  • Store hands the service over to its next instance. In the instance that stops, Store::hand_over stores the state of the service and items, such as the connections of its clients, each a record of bytes and file descriptors, however many there are. In the next instance, Store::take_over takes them over, as a Restored, and tells systemd that the service has started. A hand-over is stored whole and taken over whole, or either fails: nothing of it is left out.

A protocol takes part through two traits, Save and Restore, which the hand-overs of its connections implement, so that a service hands those over as they are. The testing feature adds what the tests of the restarts of a service need: testing::Manager, a fake service manager, and testing::store, the store of a service that a test runs in its own process.

zoter is named after Soter, the Greek daimon of safety, preservation, and deliverance from harm. It builds nothing on platforms other than Linux, where systemd is. Its API is synchronous, and depends on no async runtime.

The life of a service

use std::os::fd::OwnedFd;

use zoter::{Error, Passed, Store};

fn main() -> zoter::Result<()> {
    // The next instance takes the file descriptors of a hand-over out of the store under this
    // limit, and fails to take it over if they don't fit.
    zoter::raise_fd_limit()?;
    // SAFETY: No other thread runs yet, so none reads or writes the environment meanwhile, and
    // nothing took the file descriptors that systemd passed.
    let mut passed = unsafe { Passed::from_env() }?;

    // The store that systemd keeps for the service, which the units below have it keep, for a
    // service that systemd passed a socket. Without systemd there is none, and the example stops
    // here. A service may as well serve on without a store then, and disconnect its clients as it
    // stops.
    let mut store = match Store::from_env("example", &passed) {
        Ok(store) => store,
        Err(Error::NoManager) => return Ok(()),
        Err(e) => return Err(e),
    };
    // The file descriptors stored under `example/` are taken out of `passed`, and removed from the
    // store, and the hand-over that they hold is read, if there is one. systemd takes the service
    // for started from here on.
    let restored = store.take_over(&mut passed)?;
    // What is left in `passed` is the sockets of socket activation, by the names of their units.
    let _listener = passed.take("example.socket");
    // The clients that the previous instance handed over, each a socket here.
    let mut clients: Vec<(u64, OwnedFd)> = Vec::new();
    if let Some(mut restored) = restored {
        let _state = restored.state();
        for item in restored.take_items() {
            let id = item.id;
            clients.push((id, item.restore()?));
        }
    }

    // Serve the clients, and the ones that connect, until systemd sends `SIGTERM`.

    // Stop taking requests, finish those taken, and hand the state and all of the clients over.
    let clients = clients.iter().map(|(id, client)| (*id, client));
    store.hand_over(b"the state".to_vec(), clients)?;

    Ok(())
}

The units of a service that restarts so, example, are:

# example.socket
[Socket]
ListenStream=/run/example

[Install]
WantedBy=sockets.target
# example.service
[Unit]
# systemd otherwise starts the service without its socket where the socket unit isn't active.
Requires=example.socket

[Service]
# The service has started once it took over, which zoter tells systemd.
Type=notify
ExecStart=/usr/bin/example
# The memfd of the hand-over, and the sockets that hold its file descriptors, some 70000 in each.
FileDescriptorStoreMax=16

systemctl restart example then restarts the service with its clients, which needs systemd 254 or later. A stop, or a start after a stop, doesn't keep them: systemd closes the store as the service stops for good. Without FileDescriptorStoreMax= in the unit, or under an older systemd, there is no store, and Store::from_env fails with Error::NoStore. Without a socket of socket activation, which Requires= on the socket unit sees to, there is none either, and it fails with Error::NoSocket: the service tells systemd that it has started as it takes over, and its clients wait in the socket meanwhile.

Errors

The fallible functions of zoter return a zoter::Result. The one exception is restoring an item, which fails with the error that its Restore implementation chooses. An Error says what went wrong:

  • Error::NoManager means that systemd takes no notifications from the process, as when it doesn't run it at all. A service that also runs without systemd goes on without a store then.
  • Error::NoStore, Error::StoreTooSmall and Error::StoreFull name the unit setting to raise, FileDescriptorStoreMax=.
  • Error::NoSocket means that systemd passed the service no socket of socket activation, which needs a socket unit, and Requires= on it in the unit of the service.
  • Error::InvalidHandover and Error::UnsupportedHandover mean that the next instance can't take over the hand-over. It then starts afresh without it.
  • Error::Io is the error that the system returned, such as EMFILE when the file descriptors of a hand-over don't fit under the limit of open files.

Each variant converts into an std::io::Error of a matching kind. So a service whose functions return std::io::Result can use ? on these errors too.

Status

In development: the API can change from one minor version to the next.

License

MIT

Sponsors