Module durable 

Saving object state off the object database lock (#1270).

The Audit Log, the Notification Forwarder, the Notification Class and the Access Rights object keep state in storage the application provides (AuditLogPersistence, NotificationForwarderPersistence, NotificationClassPersistence and AccessRightsPersistence). A storage call can be slow, since the file backends write, synchronize and rename, so none of these objects makes one while its caller holds the database’s write guard. They all follow the pattern this module provides.

• Under the guard, the object copies the state to keep and queues the copy on its save writer. The writer’s own thread makes the storage calls, one at a time and in the order they were queued.
• A save whose outcome decides a request is staged. The object sets the state the request would leave aside and keeps serving the old one. The server drops the guard and waits for the save’s SaveWait on Tokio’s blocking pool, so a paused test clock does not jump while the writer thread works; then, holding the guard again, it runs the request as usual, and the object takes the saved state, or refuses the request if the save failed. So a forwarder list write that cannot be saved is refused and leaves the old list, and an Audit notification is stored, and a confirmed one acknowledged, only once its commit is durable. An application’s Audit Log purge (#1238) stages the same way, though no property is written.
• A save no request waits for, such as a forwarder’s lapse and minute saves, coalesces: a queued save the thread has not started is replaced by the newer one, so a burst costs one save of the latest state.

Storage leads the served state

A staged state reaches storage before anyone can read it from the object. The storage call returns on the writer’s thread, and the object serves the new state only when the request that staged it holds the guard again and takes it (or a later stage, or the object’s operation task, settles it). Peers never see that gap: the request answers, and a confirmed Audit notification is acknowledged, only after the object serves the saved state. Code that watches storage directly does see it. A storage call that has returned, such as an in-memory AuditLogPersistence whose commit has stored a record, does not mean a read or an AuditLogQuery finds the record yet; wait on the served state, such as the log’s Record_Count, instead.

A request that makes several writes to one object, as a WritePropertyMultiple can, stages them as one save of the state they leave together. Each write takes its own step of it as the request makes it, so the object serves the state the last write leaves only once every write has been made, and a request that stops part way keeps the steps it took while storage goes back to that served state (#1423). Under a mutation authorizer, the server asks it about such writes before staging them, and stages only the ones it allows (#1321).

Some paths still wait for a save while the guard is held. They get the same outcome through the same writer; the object just queues the save and waits for it where it is:

• a write nobody staged, such as application code writing through the database;
• an in-place change to an Audit Log, such as add_record, which first lets a staged commit land, waiting for it if it is still running.

A write nobody staged takes the place of a staged write it doesn’t match only when the object will make it: a write the object refuses leaves the staged write alone (#1424).

Dropping an object waits for the saves it has queued. The server’s DeleteObject therefore drops a removed object on a blocking thread after releasing the guard, and a server dropped without stop() in async code drops its database there too (#1409); application code that removes an object, or drops the last handle on the database, should do the same. bacnet_server::server::drop_database_off_runtime lets go of a handle on the server’s database that way, as the server’s own detached tasks do (#1513). An object dropped with a write still staged for a request that never came back first saves the state it serves, so storage never keeps a state no client was told about, and waits for that save too (#1363). The server’s stop() settles such writes once it has joined its requests ([DurableWrites::settle_forgotten_writes]) and waits for their saves, so a database dropped after a stop has nothing more to save. A server dropped without stop() returns before those saves land.

The writer’s thread is a plain std thread with no Tokio runtime, so a storage implementation that needs one brings its own handle. A storage call that panics fails that save. An object starts the thread with its first save and keeps it, parked while idle, until the object is dropped: one thread per object that has saved.

Adding an object

Any other object that keeps a written state in storage reuses this module instead of saving under the guard:

1. Own a staged::StagedSaves over a SaveWriter of a snapshot of the state to keep. It stages one save of the state a request’s writes leave (staged::steps works out each write’s step), hands each write its step or the save’s error, and drops a staged write its request never made, leaving the object to save its served state at once (StagedSaves::correct), or as the object drops. A write nobody staged saves with StagedSaves::save_now, once the object knows it will make it, and a save nobody waits for coalesces through StagedSaves::submit_coalescing.
2. Implement [DurableWrites] for the object, including settle_forgotten_writes (StagedSaves::drop_forgotten), and return it from BACnetObject::durable_writes_internal.
3. Add the object type, and the properties it saves, to may_save in the server’s durable_writes module. The server stages only the writes listed there; any other write saves in place.
4. Give a file backend a file::ObjectFile, which tags the file with the format and the object identifier, caps its size, and replaces it through a synchronized temporary file, a rename and sync_parent_dir.

The Notification Forwarder’s saving module is the full example: a list write stages, and the operation task’s saves coalesce. The Notification Class’s is the smallest: a Recipient_List write stages, and nothing else saves. The Access Rights object’s stages indexed array writes too, keyed by the index as well as the value.

Structs

SaveWait

A future that resolves once a save has run, or once a staged write that held an object has gone, whichever it was handed out for.