actordb

A Rust actor-based distributed database with pluggable, opt-in storage

• 

• 

• actordb:

Crate installation

The next release is 0.2.0 (Rust 1.88+). After it is published:

[dependencies]
actordb = "0.2"

The default crate includes no concrete storage backend. Its actor cache and KvStore contracts work without a database; NodeConfig.store remains None unless you supply a custom store. RocksDB is an independent opt-in:

actordb = { version = "0.2", features = ["rocksdb"] }

The separate memory feature exposes the no-op MemoryStore test double; it is not persistent storage. Neither backend feature enables the other. All configurations build protobuf from source; only opting into RocksDB requires its native library and libclang bindings. default-features = false is not necessary to avoid RocksDB: the default feature set is empty.

This remains pre-1.0 software, and 0.2 includes breaking API, wire-protocol, and legacy-store compatibility changes. Read the release notes and release procedure before upgrading or publishing.

Working prototype

A working, tested prototype of the actor-first sharded database vision:

The workspace requires Rust 1.88 or newer; CI pins 1.88 as the minimum supported toolchain.

This library workspace does not commit Cargo.lock; fresh checkouts resolve compatible dependency versions, preferring the declared Rust minimum. Cargo can generate an ignored local lockfile. Run cargo update to refresh it and rerun tests; dependency resolution is not frozen across fresh builds.

• shard: deterministic key → fragment mapping (fragment_of via seahash). The keyspace is partitioned into fragment_count fragments; every node agrees on the mapping.
• actor: per fragment, a KvActor dispatcher (built on ractor) routes ops to a private managed worker pool by hash(key) % worker_count. Sticky dispatch preserves per-key ordering within a priority pool while different keys execute concurrently. Workers share the fragment's FragmentState: one in-memory cache with a lock-coherent ordered key index, one ordered persistence buffer, and one KvStore; cross-pool LWW merges are serialized there.
• Supervisor actors (in actor): every fragment runs under a FragmentSupervisor owning the dispatcher pool plus background actors — a Cleaner (durable TTL-expiry markers) and a Snapshotter (periodic durable flush). Dead workers restart in place; dead children restart without touching sibling fragments. Node::health() reports per-fragment health. The supervision module preserves older imports as compatibility reexports; implementations live alongside the other actors.
• node: a Node pairs the per-fragment supervisors with a Tokio TCP mesh. Static placement (fragment_map) with caller-side one-hop routing: a node either applies a write locally and enqueues it for async replication, or forwards it to a fragment owner. Inbound ops merge with last-writer-wins on hybrid-logical-clock timestamps (tie-broken by node_id), which is idempotent and commutative, so replicas converge without coordination. Deletes are tombstones that propagate like writes. Joining and restarted nodes converge through correlated, byte-bounded reconciliation after the peer handshake.
• store: pluggable durable storage. In-memory by default; enable the rocksdb feature for RocksDB (per-fragment key namespacing via PrefixedStore, a fail-closed identity/layout manifest, exclusive live attachment, validated physical routing, and synchronous WAL writes by default). Shared contracts and options stay in store; implementations live in its memory, prefixed, and feature-gated rocksdb child modules, with store::* imports available through reexports when the corresponding backend feature is enabled. No bundled backend is enabled by default.
• proto: the wire protocol — prost-encoded Envelope frames, length-delimited over TCP.

Run the unit and real-localhost-TCP integration tests:

cargo test

Run the three-node demo (writes from each node, prints every node's converged local copy):

cargo run --example three_node_demo

Documentation

Comprehensive guides live in docs/:

• Architecture — what actordb is, crate layout, the fragment/supervisor/worker-pool hierarchy
• Public API guide — Node, NodeConfig, puts/gets/deletes, put_multi/get_multi, TTLs, priorities, health APIs, with examples
• Persistence — RocksDB backend, on-disk format, PrefixedStore, flush batching, crash semantics
• Routing & placement — key→fragment→owner, PlacementController, migration protocol, placement_report()
• Replication — sender/receiver state machine, sessions, acks, byte-bounded reconciliation, and the protocol-v3 handshake
• Tombstones & deletes — delete lifecycle, conflict rules, retention, and requirements for safe future reclamation
• Consistency model — LWW via HLC, exact local apply outcomes, read-your-write, partitions
• Multi-key batching — fan-out, failover, partial failures
• Traffic classes & backpressure — OpPriority, pool config, shed behavior and observability
• Configuration reference — every NodeConfig field with defaults and guidance
• Operations — crash/restart, rolling restart, partitions, overload, what to monitor
• Test strategy — unit vs integration vs Docker scenarios
• Releases — package checks, version tags, and crates.io publication

Docker integration tests

Modeled on ractor's cluster integration tests: three nodes on a private compose network (test-net), each container running one scripted role from the actordb-docker-test binary (docker-tests/ crate). Unlike the in-process tests, these exercise real multi-host networking — DNS peer addresses, container death, and true SIGKILL crashes.

Scenarios live in docker/envs/:

• replication — node-a writes one key; node-b and node-c must converge on it.
• restart — SIGKILL node-a mid-cluster (a real kill -9, delivered by the driver script), restart it, and require all three nodes to reconverge. Exercises the redial supervisor, fresh replication sessions, and immediate correlated reconciliation across process death.

docker compose build
bash docker/run-scenario.sh docker/envs/replication.env
bash docker/restart-test.sh

CI runs both scenarios on every push/PR touching the relevant paths (.github/workflows/docker-integration.yaml).

Public API

actordb::node — the main entry point

use actordb::node::{Node, NodeConfig, ReplicationDrainOptions};
use std::time::Duration;

let node = Node::start(
    "node-a".to_string(),
    "0.0.0.0:7447".parse().unwrap(),
    NodeConfig::builder().build(),
)
.await?;
// port 0 picks an ephemeral port; the bound address is available via `node.addr()`

node.connect("127.0.0.1:7447".parse().unwrap()).await?;
// connect() returns after the transport dial is owned by the redial supervisor;
// authentication and immediate reconciliation continue asynchronously. The
// first authenticated peer id at this address is pinned for later reconnects.

node.put("key".to_string(), b"value".to_vec()).await?;
node.put_with_ttl("temp".to_string(), b"value".to_vec(), Some(Duration::from_secs(60))).await?;
node.delete("key".to_string()).await?;

let value = node.get("key".to_string()).await?;  // routed read, Option<VersionedValue>
let all = node.get_all().await?;                  // full local copy (live keys)
let health = node.health().await?;                // per-fragment supervision health

// Bulk APIs: keys hash to fragments, then exact-byte/count-bounded chunks per
// remote fragment/owner attempt. Per-key Results remain in input order.
let puts = node.put_multi(vec![
    ("k1".to_string(), b"v1".to_vec()),
    ("k2".to_string(), b"v2".to_vec()),
]).await;
let gets = node.get_multi(vec!["k1".to_string(), "k2".to_string()]).await;
let peers = node.peer_count().await;

// Local-only: prompt even when a replica is unavailable.
node.shutdown().await?;

// Alternatively, choose the bounded peer fence before local shutdown:
// let report = node.shutdown_with_replication_drain(
//     ReplicationDrainOptions::builder().timeout(Duration::from_secs(30)).build(),
// ).await?;

Node::start uses the default transport and replication liveness policy. Use Node::start_with_options when those runtime-only limits need tuning:

use actordb::node::NodeRuntimeOptions;
use std::time::Duration;

let runtime = NodeRuntimeOptions::builder()
    .peer_queue_frame_capacity(512)
    .peer_queue_byte_capacity(64 * 1024 * 1024)
    .replication_control_message_capacity(128)
    .inbound_connection_capacity(512)
    .receive_byte_capacity(128 * 1024 * 1024)
    .queue_admission_timeout(Duration::from_secs(2))
    .reconcile_data_admission_timeout(Duration::from_secs(20))
    .forwarded_attempt_timeout(Duration::from_secs(8))
    .connect_timeout(Duration::from_secs(3))
    .handshake_timeout(Duration::from_secs(3))
    .socket_read_timeout(Duration::from_secs(20))
    .socket_write_timeout(Duration::from_secs(20))
    .replication_ack_timeout(Duration::from_secs(20))
    .build(); // Omitted relay, read, mutation, and health settings retain defaults.

let node = Node::start_with_options(
    "node-a".to_string(),
    "0.0.0.0:7447".parse().unwrap(),
    NodeConfig::builder().build(),
    runtime,
)
.await?;

All configuration and actor argument types provide Bon builders with defaults for optional tuning and compile-time checks for required dependencies. Existing struct literals, Default, with_* helpers, and getters remain available. build() constructs settings; startup/spawn still validates their values. See the builder reference for nested pool defaults, optional setters, and the 13 supported input types.

The defaults are a 256-frame queue per peer, 64 protocol-control messages per fragment/peer sender, at most 1024 concurrent inbound handshakes or connections, and a node-wide 64 MiB raw receive-body budget. Accepted and outbound-dialed connections reserve body bytes before allocation, including handshake frames; saturation closes only the contributing connection. Permits remain attached through decoding and the handler turn. This counts encoded bodies, not protobuf/actor/cache copies or total process memory. Queue admission, a complete forwarded owner attempt, one dial, and the absolute two-flight handshake each have 5-second deadlines. Reading a frame body after its length prefix and writing a complete frame each have a 30-second deadline; an idle established connection has no read deadline until it sends a prefix. Streamed reconciliation-data pages have a separate 30-second queue-admission deadline so a legal near-limit value can wait behind another slow near-limit frame; an acceptable ordinary replication ACK is due within 30 seconds. Locally applied mutations have a separate per-sender ingress lane sized by SenderConfig::buffer_capacity; saturation schedules lossless reset reconciliation without blocking the write path. Reconciliation barriers use SenderConfig::reconcile_timeout instead of the ordinary ACK timeout. Anti-entropy and ordinary replication frames are built against their exact encoded size, targeting 1 MiB pages and never exceeding the 16 MiB hard frame limit. The peer writer also enforces a 32 MiB default encoded-byte budget in addition to its frame-count limit.

Multi-key semantics:

• put_multi / get_multi partition keys by fragment and send one or more exact-byte-bounded BatchPut / BatchGet requests per remote fragment and owner attempt, rather than one message per key. Large get responses advance as connection-correlated ordered prefixes; omitted suffixes are requested under fresh ids without losing duplicate positions.
• Local fan-out is capped at 1024 concurrent calls before dispatcher admission. A remote BatchGet uses one aggregate actor continuation but admits and sticky-routes only its next ordered key, stopping by exact encoded result bytes before an omitted suffix is reserved or cloned.
• Local and remote puts await the serving fragment's exact per-key apply outcome. Remote batches preserve request order/cardinality and use the configured per-attempt deadline (5 seconds by default), including outbound queueing and the response.
• Remotely routed single-key and batch operations fail over across replica holders in placement order on transport, timeout, or malformed-response failures. Every owner attempt has one configured budget (5 seconds by default) covering queue admission and the response. Typed Unavailable positions try the next holder with their original stamped mutations, even after other positions in the batch finish. Other valid per-key outcomes are returned directly and are excluded from subsequent retries.
• Errors are per-key and classified: NodeError::Actor for misconfiguration (no replica listed for a fragment) or local actor-framework failure, NodeError::Network for connectivity failures, plus exact Backpressure, Storage, Superseded, and Expired mutation outcomes.
• put_multi_with_priority / get_multi_with_priority set the operation priority explicitly (default: Background); the priority is carried on the wire and honored by the remote owner.

Key NodeConfig knobs:

The local node id, every members key, and every placement replica id must contain 1 to 255 UTF-8 bytes. Placement replica lists must be duplicate-free. RocksDB permanently binds its first Node id and fragment count in a separate metadata column family and rejects mismatches before network bind. Non-empty pre-manifest databases are rejected without mutating their column-family layout and require explicit one-time legacy adoption; see Persistence.

• Local and remote writes await the serving worker's exact apply outcome; after Ok, that owner reads the version or a newer winner. Replication to the owner's other holders remains asynchronous.
• Conflicts resolve with last-writer-wins on HLC timestamps, tie-broken by node_id; tombstones win exact ties.
• Failures surface as typed NodeError variants, including Backpressure, Storage, Superseded, and Expired rather than false success.

actordb::actor — actors and the storage core

• KvActor — public per-fragment dispatcher; its managed sticky-hash worker pools are internal; FragmentState — the shared cache/buffer/store. Direct embedders send KvMessage to KvActor rather than splitting admission from a raw worker route.
• VersionedValue { value, timestamp_ms, node_id, expires_at_ms, is_tombstone } — a value plus its LWW version; wins_over implements the conflict rule, is_expired the TTL check.
• HlcClock — hybrid logical clock behind all timestamps.
• ApplyOutcome::{Applied, AlreadyPresent, Superseded, Expired} — the exact result used instead of inferring write success from a later read.
• ApplyError::{Backpressure, InvalidInput, Storage} distinguishes a nonfatal unencodable direct actor request from persistence failure; invalid input does not restart the fragment worker.
• BatchReadEntry, BatchReadPage, SnapshotSummary, SummaryPage, and VersionedPage support priority-aware ordered batch reads plus pinned-time, revision-aware anti-entropy reads without cloning whole fragments or arbitrary fetch chunks; existing GetAll and GetSnapshot remain available for compatibility and debugging.
• KvMessage::{ApplyOp, Flush, FlushStable, Get, GetBatch, GetAll, GetSnapshot, GetSummaryPage, GetVersioned, GetVersionedPage, GetRevision, GetShedCounts} — for embedders driving the dispatcher directly instead of through Node. KvMessage is non-exhaustive; downstream matches must retain a wildcard arm as internal routing/query variants evolve. FlushStable requests the backend's explicit stable fence and reports ApplyDurability::StableCommitted only when that fence succeeds.

Traffic isolation + backpressure (Phase 7)

Each fragment runs three isolated worker pools instead of one:

• OpPriority::{High, Default, Background} selects the pool. put_with_priority / get_with_priority (and the put_multi_with_priority / get_multi_with_priority variants) set it; plain put / get / put_multi / get_multi use Default (single-key) or Background (bulk).
• Each PoolConfig sets workers, queue_cap, and an optional token-bucket rate_limit_per_sec. When a pool is over its queue cap or rate limit, the op is shed explicitly: counted in the pool's shed counter, and the caller gets NodeError::Backpressure — a shed Get never returns Ok(None), so backpressure is never confused with "not found".
• queue_cap covers work accepted into that priority's workers (active plus queued), not the shared dispatcher mailbox or replicated ingress. Accepted reads are handed to their worker without making the dispatcher await them, so a blocked background worker does not block high-priority admission, health, or supervision.
• Priority is preserved end-to-end: forwarded single-key operations and batch messages carry the originator's priority on the wire, and the owner applies/serves them in the matching pool. Priority selects that remote actor pool; it does not prioritize the per-peer socket's ordinary data lane.
• Shed counts are surfaced per pool (high, default, background) via node.health() → FragmentHealth.shed_counts.

Ordering honesty: local and forwarded puts wait for the serving worker's exact apply outcome. At that point its shared cache contains the applied version or an identical replay; by return time a newer concurrent LWW winner may exist. The configured remote deadline (5 seconds by default) covers both bounded outbound-queue admission and the reply; cancellation and late replies remove or ignore pending state.

Supervisor and maintenance actors

These types live under actordb::actor. actordb::supervision reexports the same types for compatibility, including their messages, state, and Bon builders.

• FragmentSupervisor — owns a fragment's dispatcher, Cleaner, and Snapshotter; restarts dead children in place.
• FragmentHealth — per-fragment health snapshot (worker count/names, restart counts, per-pool shed_counts).
• Cleaner — materializes expired values as durable TTL markers.
• Snapshotter — periodically flushes the fragment's retained write buffer.

actordb::replication — async sender/receiver

• Per fragment, per replica peer, a sender task batches mutations (writes + tombstones) off the write path: ships on a timer, bounded in-flight, and retains each batch until a typed ACK reaches the configured durability fence. Receiver apply/flush failures NACK and requeue the exact batch.
• An ordinary ACK has a 30-second default deadline. Expiry rotates the sender session and replays all unacknowledged, already-stamped operations in their original order; LWW makes duplicates idempotent, and ACKs from the old session are ignored. An ACK below the requested durability neither retires the batch nor extends its deadline.
• Session ids per pairing; stale batches discarded on reconnect.
• Buffer overflow → buffer_reset: shed, rotate the session, and reconcile via timestamp/tombstone comparison immediately or when the peer returns.
• Periodic anti-entropy repairs residual divergence in both static and dynamic placement. Both sides stream ordered, byte-bounded summary pages at a pinned TTL time; revision drift or any repair forces a fresh stable verification pass. Fetch requests contain up to 256 exact-byte-bounded keys; the responder reads values incrementally and streams byte-bounded data pages through its bounded writer. Repair and ordinary replication batches use the same exact protobuf sizing: 1 MiB soft pages and 16 MiB hard frames. Reconcile rounds are counted per sender (SenderStateReport::reconcile_rounds).

actordb::placement — dynamic placement controller (Phase 8)

Opt-in via NodeConfig::placement_controller. With no controller the node keeps static placement and behaves exactly as before.

use actordb::placement::ManualPlacement;
use std::sync::Arc;

let controller = Arc::new(ManualPlacement::new(initial_map));
let config = NodeConfig::builder()
    .placement_controller(controller.clone())
    .placement_poll_interval(Duration::from_millis(500))
    .build();
// ... later, migrate fragment 0 from "a" to "b":
let mut overlap = HashMap::new();
overlap.insert(FragmentId(0), vec!["a".to_string(), "b".to_string()]);
overlap.insert(FragmentId(1), vec!["a".to_string()]);
controller.set_placement(overlap);
// wait until b reports fragment 0 as Serving...
let mut final_map = HashMap::new();
final_map.insert(FragmentId(0), vec!["b".to_string()]);
final_map.insert(FragmentId(1), vec!["a".to_string()]);
controller.set_placement(final_map);

• PlacementController trait: desired_placement(), on_membership_change(), on_fragment_ready(). Implement it for an external cluster manager; ManualPlacement (interior-mutable map) covers tests and operator-driven moves.
• The driver polls the controller, publishes its map as the routing table, and converges: gained fragments get a supervised subtree plus a streamed bootstrap from every connected source candidate (on-demand reconcile with bounded fetch chunks and value pages, 30s timeout); only an already-serving holder may answer, and timeout leaves the new holder non-authoritative. Revoked fragments drain their replication senders, then stop.
• Dynamic startup synchronously loads assigned durable subtrees for validation but leaves them Bootstrapping until an already-serving replica completes a verified reconcile round. For a brand-new cluster, seed each fragment on one node first, wait for Serving, and only then publish overlap replicas; simultaneously starting an all-new replica set remains non-authoritative by design because no node can prove it has the latest state.
• Migration protocol (controller-sequenced, with overlap): add the new owner → wait for it to report Serving (bootstrap done) → revoke the old owner. At least one node always serves the fragment; LWW keeps the overlap consistent. A bootstrapping node forwards reads/writes to the previous owner until it flips to serving.
• A node that receives a routed op for a fragment it no longer serves re-forwards it to the current owner (hop-limited, never bounced back to the sender) instead of dropping it — this covers the window where peers' placement maps disagree.
• node.placement_report() — per-fragment lifecycle state (Bootstrapping / Serving / Draining / Stopped) plus the authoritative placement map.
• Static and dynamic placement use the same correlated reconciliation engine. SyncRequest/SyncResponse are retired in protocol v3, and node.sync_entries_applied() is a compatibility counter that remains zero.
• Assumes the peer mesh is established (members or manual connect()); the driver does not dial.

actordb::proto — wire protocol

Prost-generated protobuf types (Envelope, Hello, HelloAuth, ReplicateOp, ForwardedPut, ForwardedGet, GetResponse, ReconcileRequest, ReconcileResponse, ReconcileFetch, ReconcileData, SyncEntry, BatchPut, BatchGet, BatchGetResponse, MutationResponse, BatchAck, BatchNack), sent as 4-byte length-prefixed frames over TCP. SyncRequest and SyncResponse remain generated for source compatibility but are retired; a negotiated v3 peer that sends either is rejected. These types are mostly of interest when implementing a compatible peer.

Current limitations

This is a working prototype, not production-ready. Known gaps, tracked as issues in the issue tracker:

• Persistence is opt-in — pass a store in NodeConfig for durability; without it the dataset lives in memory and a restart wipes it (recoverable only via peer reconciliation).
• No built-in TLS or individual peer identity — plaintext TCP. Protocol v3 gives each physical connection two fresh 256-bit challenges. With a NodeConfig::cluster_secret containing at least 32 independently generated, high-entropy random bytes, both peers prove fresh possession by HMAC-SHA256 over the ordered dialer/listener transcript, roles, ids, version ranges, capabilities, and challenges. This prevents replay of a captured v3 handshake, but a shared secret proves cluster membership, not which member is speaking: any holder can claim any node id. A guessable password can also be tested offline against a captured transcript. The proof provides no confidentiality or integrity for application frames after the handshake. Use a protected, mutually authenticated custom Network when those properties matter.
• Anti-entropy is correlated reconciliation — static placement, dynamic placement, late joins, reconnect recovery, buffer resets, and periodic repair all use the same session/round-correlated reconciliation engine. In legacy map-empty mode, per-peer senders are reaped on disconnect instead of retaining an unbounded set of discovered identities; mandatory immediate reconciliation after reconnect recovers downtime writes and tombstones. Senders named by an explicit placement map remain allocated while disconnected and retain their bounded replay/reset state.
• Transport queues have count and byte bounds — the shared per-peer writer pipeline defaults to 256 frames and 32 MiB of admitted encoded data; a single accepted frame can approach 16 MiB. A producer first reserves the data-lane ingress and its exact encoded-byte permits, then allocates the frame; the owned permit travels with the frame through the socket write and releases on every drop path. Queue admission and socket writes have bounded waits; expiry cancels the connection so the dialer can reconnect. The frame budget is split between data and a writer-priority response lane, and a full 16 MiB maximum response is reserved from ordinary producers. Responses produced by the sole connection reader use nonblocking admission and cancel on saturation, so the reader never waits behind its own writer. After at most 16 consecutive responses, a ready data frame is written to prevent starvation. Non-serving read/mutation relays separately share node-wide limits of 1024 live attempts/timers and 64 MiB of retained request-key storage (String/Vec storage and buffer capacities, not mutation values or total heap). Admission never waits; rejected mutations return Unavailable, and legacy reads stay unavailable without false misses. Exact physical origin/owner identities prevent stale disconnect cleanup from removing replacement work. Each Node sender has a bounded mutation lane and a separate bounded protocol-control lane; mutation overflow advances a persistent reset generation, while control overflow closes the contributing connection for replay. Sender actor lanes remain operation/message-count bounds. Protocol builders separately enforce exact encoded sizes: anti-entropy, ordinary replication, and aggregate batches target 1 MiB pages and no frame may exceed 16 MiB. Mutation batches also reserve their worst-case complete replies; get responses use ordered prefix pulls. An irreducible oversized record or a record with empty or over-limit provenance is rejected before local apply; a legacy persisted record violating either invariant fails node startup instead of becoming permanently unreplicable. Before Prost decoding, an allocation-free scan caps anti-entropy pages at 256 repeated items and general batch fields at 4096, preventing empty-message decode amplification. One ticketed FIFO credit schedules anti-entropy rounds fairly across every fragment on a physical connection. Receiver actor mailbox byte accounting remains follow-up work. Inbound handlers are capped at 1024 by default, the two-flight handshake has one absolute 5-second deadline, and completed Node-owned tasks are reaped while the node runs.
• Fan-out is sequential — remote fragment groups and their replica-holder attempts are currently awaited one at a time. Operation priority chooses the destination actor pool but does not schedule the ordinary socket data lane.
• Wire upgrades are negotiated but coordinated — protocol-v3 Hello advertises an inclusive version range, required capability bits, and a fresh challenge; HelloAuth completes the transcript proof. Missing or incompatible fields fail closed before peer installation. The current binary supports only wire v3 and requires the paged-anti-entropy, byte-bounded-batches, and stable-durability-ACK capabilities. Byte-bounded batches change batch-get responses to ordered-prefix semantics, while stable durability ACKs are required before RocksDB may retire a handed-off prefix; binaries missing either contract are rejected. It is incompatible with v2, so deploy compatible binaries together rather than performing a mixed v2/v3 rolling upgrade. This transition is also a Rust source break for advanced users of the public generated actordb::proto API and the public replication::SenderMsg: reconciliation messages gained cursor, revision, page, and completion fields, so struct literals and exhaustive assumptions must be updated.
• Shutdown is explicit and fallible — Node::shutdown().await atomically closes public-operation admission (NodeError::ShuttingDown for later calls), joins listener/connection/dial work, waits for already-admitted API work, drains the bounded serving forwarded-mutation completion owner without aborting accepted apply/outcome/fanout, stops replication senders, drains every fragment mailbox, and reports the final local-store flush. A flush error retains the quiesced fragment and its batch; retry shutdown until it succeeds. The node does not resume after an error. This local fence does not wait for replication ACKs. shutdown_with_replication_drain is the opt-in bounded alternative: it freezes placement, waits for admitted mutations, and requires a stable full reconciliation plus the configured ACK durability from every frozen target before entering the same local shutdown. Timeout/cancellation leaves the quiesced node and transports alive for retry; ordinary shutdown explicitly abandons the peer guarantee. RetryableSender requests a fresh exact sender fence, while Unavailable means local-only shutdown has already made the guarantee impossible. Its report proves a per-target fenced LWW version floor, not exact/global cache equality, quorum, or a backend-independent fsync guarantee. Offered RocksDB defaults to SyncWal; explicit AsyncWal is weaker for StoreCommitted, while StableCommitted forces the explicit WAL fence. Legacy empty-map mode covers only senders known at the fence. Dropping Node only requests best-effort cancellation and stop; it cannot report a flush failure, await a join, or safely release the same live RocksDB attachment. This pre-release change is a Rust source break: callers must handle the new shutdown Result, and exhaustive matches over NodeError, ReplicationAckDurability, ApplyDurability, KvMessage, the generated ReplicationAckLevel, or the advanced SupervisorMessage API must include their new variants (StableCommitted and FlushStable where applicable).
• Tombstones are retained — Node::start rejects every tombstone_gc_age: Some(..) setting. Safe reclamation needs durable per-replica convergence evidence; elapsed wall-clock time cannot prove that condition. Whole retired fragment namespaces are a separate lifecycle case.
• Revoked RocksDB prefixes retire after a stable revision-pinned handoff — placement seals every replacement sender, requires every current replica to complete a fresh full reconciliation and StableCommitted barrier at one unchanged source revision, then quiesces and flushes the source before RocksDB commits a synchronous idempotent range tombstone. A weak ACK, replica-set/generation change, fetched winner, deletion error, or re-add fails closed in Draining or retains the namespace. Custom stores retain prefixes unless they explicitly opt into lifecycle deletion. RocksDB compaction owns later physical space reclamation. The certificate covers LWW winners and their TTL metadata; independent expiry cleanup may make later byte snapshots differ.
• Custom worker-pool supervision is not lossless — a worker crash drops accepted jobs in that worker's mailbox and closes their reply ports; those jobs are not replayed. Exact per-worker reservations reclaim their capacity, and orderly shutdown drains accepted work. A restarted dispatcher is republished through a generation-checked fragment route, but migration to ractor's Factory lifecycle remains tracked for discard policy and lossless restart. The formerly public split WorkerPool admission/routing surface was removed in this pre-release API: direct embedders should use KvActor.

Contributors

To learn more about contributing to actordb please see CONTRIBUTING.md.

License

This project is licensed under MIT.