Skip to main content

Crate quepaxa_ffi

Crate quepaxa_ffi 

Source
Expand description

Stable C ABI over the quepaxa consensus core, for C, C++, Go, Swift, Kotlin/JNI, Python (ctypes/cffi) or any language that can call C.

The ABI exposes the same sans-IO quepaxa::Replica: your program feeds it events and carries out the actions it returns. Read quepaxa::guide::driving for the model; this page covers what is specific to C.

§Getting the library

  • Prebuilt: every GitHub release attaches quepaxa-ffi-<version>-<target> archives with include/quepaxa.h and the static and shared libraries for Linux (x86_64, aarch64; glibc 2.28+), macOS (x86_64, arm64), Windows (x64, arm64, MSVC), Android (jniLibs for four ABIs, 16 KB pages) and iOS (an XCFramework).
  • From source: cargo build --release -p quepaxa-ffi produces libquepaxa_ffi.a / .so / .dylib (or quepaxa_ffi.lib / .dll) in target/release; the header is crates/quepaxa-ffi/include/quepaxa.h.
  • Linking statically on Linux also needs -lpthread -ldl -lm.

§The loop in C

#include "quepaxa.h"

uint32_t members[3] = {0, 1, 2};
uint8_t seed[32];
fill_with_csprng(seed, sizeof seed);         /* fresh on every start */
qpx_config *cfg;
check(qpx_config_new((const uint8_t *)"my-cluster-0001!", 0, members, 3, seed, &cfg));
check(qpx_config_set(cfg, QPX_CONFIG_MAX_IN_FLIGHT_SLOTS, 32)); /* optional settings */

qpx_replica *r;
check(qpx_replica_new(cfg, records, n_records, now_ns(), &r)); /* durable records, in order */
qpx_config_free(cfg);                        /* the replica keeps its own copy */

for (;;) {                                   /* after every call: drain */
  qpx_action a;
  int32_t rc = qpx_poll(r, &a);
  if (rc == QPX_EMPTY) break;
  switch (a.kind) {
  case QPX_ACTION_PERSIST:   /* qpx_action_record(r, k, ...) for k < a.count: append + fsync,  */
                             /* then send qpx_action_then(r, k, ...) for k < a.then_count,     */
                             /* then, after draining, qpx_persisted(r, now, a.id)              */
  case QPX_ACTION_SEND:      /* send a.bytes to a.peer                                         */
  case QPX_ACTION_COMMIT:    /* apply qpx_action_entry(r, k, ...) for k < a.count, slot a.slot */
  case QPX_ACTION_COMMITTED: /* answer the client of proposal a.id                             */
  default: break;            /* new kinds may appear in minor versions                         */
  }
}
uint64_t deadline;
if (qpx_next_deadline(r, &deadline) == QPX_OK) { /* arm a timer; then qpx_handle_timeout */ }

examples/smoke.c in the crate is a complete three-replica program; CI runs it on Linux, macOS, Windows, an Android emulator and an iOS simulator.

§Contract

(Also stated in include/quepaxa.h.)

  • Every function returns a QPX_* status (0 = ok, positive = informational, negative = error) unless stated otherwise. qpx_status_str names it and qpx_last_error describes the most recent failing call on the calling thread.
  • Draining: after any mutating call, call qpx_poll until it returns QPX_EMPTY before the next mutating call (QPX_ERR_NOT_DRAINED otherwise). qpx_persisted is itself a mutating call: collect persist ids while draining and acknowledge the highest one afterwards (it covers every earlier persist).
  • Persist before send: never send a persist’s then messages before its records are durable.
  • Borrowed memory: qpx_bytes returned by qpx_poll and the qpx_action_* accessors borrow from the handle and stay valid until the next call on that handle; copy what you keep. Inputs are copied by the library; you keep ownership.
  • Threads: a handle may move between threads, but calls on one handle must not overlap. Different handles are independent.
  • Panics never cross the boundary: a panic inside the library returns QPX_ERR_PANIC and poisons the handle; every later call on it returns QPX_ERR_POISONED. Free it with qpx_replica_free.

§Versioning

qpx_abi_version returns (major << 16) | minor; compare it with QPX_ABI_VERSION_MAJOR/MINOR from the header you compiled against. Settings and metrics are not structs: the configuration is an opaque object set option by option (qpx_config_set) and metrics are read one by one (qpx_metric), so a new one is a new constant, and an option or metric the library does not know is an explicit QPX_ERR_INVALID_ARGUMENT. The only structs are outputs with a fixed layout (qpx_bytes, qpx_action, qpx_entry). Before 1.0 a minor ABI bump may be incompatible; the changelog says when.

Structs§

qpx_action
One action. Borrowed fields stay valid until the next call on the handle.
qpx_bytes
A borrowed byte range. ptr may be null when len is 0.
qpx_config
Replica configuration: create it with qpx_config_new, adjust it with qpx_config_set and qpx_config_set_leader_order, pass it to qpx_replica_new (which copies it), and release it with qpx_config_free.
qpx_entry
A committed entry.
qpx_replica
An opaque replica handle.

Constants§

QPX_ABI_VERSION_MAJOR
ABI major version: incompatible changes bump it. While it is 0 the ABI is unstable and every minor bump may be incompatible.
QPX_ABI_VERSION_MINOR
ABI minor version: additive changes bump it (any change while the major is 0). 0.2: qpx_install_snapshot may return QPX_IGNORED. 0.3: size-aware qpx_config_init, qpx_get_metrics, qpx_last_error, strict draining. 0.4: qpx_config.leader_order (appended; older layouts keep the default order). 0.5: membership changes: qpx_propose_reconfiguration, QPX_ACTION_RECONFIGURED, qpx_action_member, QPX_ERR_RECONFIGURATION_PENDING. 0.6: QPX_ACTION_RETIRED. 0.7: opaque qpx_config (qpx_config_new, qpx_config_set, qpx_config_set_leader_order, qpx_config_free) and qpx_metric replace the qpx_config and qpx_metrics structs, qpx_config_init and qpx_get_metrics (incompatible). 0.8: QPX_ERR_REQUEST_ID_EXHAUSTED; read ids use a separate range; QPX_METRIC_STEPPED_ASIDE. 0.9: QPX_DROP_SUPERSEDED, QPX_DROP_UNRESOLVED. 0.10: leader placement: QPX_CONFIG_PLACEMENT*, QPX_METRIC_SUSPICIONS, QPX_METRIC_PLACEMENT_MOVES, QPX_METRIC_EPOCH, QPX_METRIC_LEADER_POSITION.
QPX_ACTION_COMMIT
Apply the count entries of slot (read them with qpx_action_entry).
QPX_ACTION_COMMITTED
Local proposal id committed at or before slot.
QPX_ACTION_DROPPED
Local proposal id will not be reported again (flags = QPX_DROP_*). With QPX_DROP_FENCED the outcome is unknown: it may still commit, as after a crash.
QPX_ACTION_PERSIST
Append records durably after earlier persists, then send each then message, then call qpx_persisted(id). If QPX_PERSIST_REPLACE is set in flags, atomically replace all earlier records instead of appending.
QPX_ACTION_READ_READY
Read barrier id is ready once commits up to slot are applied.
QPX_ACTION_RECONFIGURED
A membership change was applied: from slot slot, the count members (read them with qpx_action_member, preferred leaders first) form epoch id (ABI 0.5).
QPX_ACTION_RETIRED
This replica is not a member of epoch id, and a majority of that epoch’s members durably know every slot before it: it may shut down, unless a later change added it back (ABI 0.6).
QPX_ACTION_SEND
Send bytes to peer.
QPX_ACTION_SNAPSHOT_NEEDED
Fetch peer’s application snapshot covering slot and call qpx_install_snapshot.
QPX_BUFFER_TOO_SMALL
The output buffer is too small; the required length was written.
QPX_CONFIG_HEDGE_MAX_NS
Upper bound of the hedge delay, nanoseconds.
QPX_CONFIG_HEDGE_MIN_NS
Lower bound of the hedge delay, nanoseconds.
QPX_CONFIG_MAX_BATCH_BYTES
Largest batch of entries per slot, bytes (u32).
QPX_CONFIG_MAX_IN_FLIGHT_SLOTS
Concurrent slots driven by this replica (u32, at most 64).
QPX_CONFIG_MAX_MESSAGE_BYTES
Largest accepted message, bytes (u32).
QPX_CONFIG_MAX_OUTSTANDING_PROPOSALS
Outstanding proposals before QPX_ERR_BUSY (u32).
QPX_CONFIG_PLACEMENT
Leader placement on this member: 1 on, 0 off (the default). Enable it on every member (ABI 0.10).
QPX_CONFIG_PLACEMENT_HOLD_NS
How long placement leaves any change of the leader order alone, nanoseconds (ABI 0.10).
QPX_CONFIG_PLACEMENT_SUSTAIN_NS
How long a member must keep a 20% gain before placement moves the leader to it, nanoseconds (ABI 0.10).
QPX_CONFIG_RETRANSMIT_INTERVAL_NS
Record request retransmission interval, nanoseconds.
QPX_DROP_FENCED
flags of QPX_ACTION_DROPPED: the replica is fenced (outcome unknown).
QPX_DROP_SUPERSEDED
flags of QPX_ACTION_DROPPED: a membership change was decided but ignored, because another change or a rotation came first. Nothing changed (ABI 0.9).
QPX_DROP_UNRESOLVED
flags of QPX_ACTION_DROPPED: a membership change was decided, but an installed snapshot hid whether it was applied; check the members in force (ABI 0.9).
QPX_EMPTY
No action, deadline or value to report.
QPX_ERR_BUSY
Too many outstanding proposals.
QPX_ERR_CORRUPT_RECORD
A durable record passed to qpx_replica_new is corrupt.
QPX_ERR_FENCED
The replica fenced itself after a storage failure.
QPX_ERR_INVALID_ARGUMENT
A null pointer, unknown option or metric, out-of-range value or index.
QPX_ERR_INVALID_CONFIG
The configuration is invalid.
QPX_ERR_INVALID_CORE_STATE
Malformed core state for qpx_compact / qpx_install_snapshot.
QPX_ERR_NOT_DRAINED
Actions from the previous call were not drained.
QPX_ERR_PANIC
The library panicked; the handle is now poisoned.
QPX_ERR_PAYLOAD_TOO_LARGE
The payload does not fit in one batch.
QPX_ERR_POISONED
The handle panicked earlier and is unusable.
QPX_ERR_RECONFIGURATION_PENDING
A membership change has not taken effect yet (ABI 0.5).
QPX_ERR_REQUEST_ID_EXHAUSTED
Request ids exhausted in this incarnation; restart from durable records (ABI 0.8).
QPX_ERR_UNKNOWN_PERSIST_ID
Unknown or repeated persist id.
QPX_IGNORED
The call had no effect: the snapshot is older than what the replica already emitted.
QPX_METRIC_CATCH_UP_RETRIES
Catch-up requests re-sent.
QPX_METRIC_CORRUPT_BATCHES
Decided batches that failed to decode.
QPX_METRIC_EPOCH
Gauge: membership epoch in force (ABI 0.10).
QPX_METRIC_FAST_PATH_DECISIONS
Own decisions on the leader fast path.
QPX_METRIC_FORWARDED
Batches forwarded to the leader.
QPX_METRIC_HEDGED
Own entries proposed directly after the leader was too slow.
QPX_METRIC_LEADER_POSITION
Gauge: this replica’s position in the leader order in force, 0 when it leads (ABI 0.10).
QPX_METRIC_LIVE_REGISTERS
Gauge: recorder registers held in memory.
QPX_METRIC_MESSAGES_REJECTED
Network messages rejected (malformed, foreign or from non-members).
QPX_METRIC_PLACEMENT_MOVES
Leader-order changes leader placement proposed from this replica (ABI 0.10).
QPX_METRIC_RETAINED_SLOTS
Gauge: decided slots held in memory.
QPX_METRIC_RETRANSMITS
Record requests re-sent.
QPX_METRIC_SLOTS_DECIDED
Slots learned as decided.
QPX_METRIC_STEPPED_ASIDE
Times this replica stepped aside because a write waited too long on its disk.
QPX_METRIC_SUSPICIONS
Times this replica suspected the member it hands entries to of being silent (ABI 0.10).
QPX_OK
Success.
QPX_PERSIST_REPLACE
flags bit of QPX_ACTION_PERSIST: replace all earlier records.

Functions§

qpx_abi_version
(major << 16) | minor.
qpx_action_entry⚠
Entry i of the current commit action.
qpx_action_member⚠
Member i of the current QPX_ACTION_RECONFIGURED action (preferred leaders first).
qpx_action_record⚠
Record i of the current persist action.
qpx_action_then⚠
Message i gated on the current persist action, and its destination.
qpx_commit_index⚠
Number of contiguous decided slots.
qpx_compact⚠
Discards log state up to through, snapshotted by the application with state.
qpx_config_free⚠
Releases a configuration (null is ignored). Replicas created from it are unaffected.
qpx_config_new⚠
Creates a configuration with default settings: cluster_id (16 bytes; messages from other clusters are dropped), this replica’s id, the members_len configured members at members (including this one; a joining replica lists the original members), and seed (32 bytes of fresh CSPRNG output on every start; never reuse it).
qpx_config_set⚠
Sets option option (QPX_CONFIG_*) to value. An unknown option, or a value too large for it, is QPX_ERR_INVALID_ARGUMENT; whether the settings are consistent is checked by qpx_replica_new (QPX_ERR_INVALID_CONFIG).
qpx_config_set_leader_order⚠
Sets the preferred leaders, first to last: every member once, or none (len 0) for “by id”. Every member must use the same order.
qpx_core_state⚠
Copies the replica’s core state into buf. Two-call pattern: with too small a cap it returns QPX_BUFFER_TOO_SMALL after writing the needed size to out_len.
qpx_handle_message⚠
Feeds a network message exactly as received.
qpx_handle_timeout⚠
Fires timers due at now_ns.
qpx_install_snapshot⚠
Adopts a snapshot covering slots up to through; QPX_IGNORED if the replica already emitted past it (do not rewind the application state in that case).
qpx_last_error⚠
Copies the description of the calling thread’s most recent failing call (empty if the last call succeeded) as a NUL-terminated string. Two-call pattern: with too small a cap it returns QPX_BUFFER_TOO_SMALL after writing the needed size (including the NUL) to out_len.
qpx_metric⚠
Reads counter or gauge metric (QPX_METRIC_*, see the Rust Metrics) into out; an unknown metric is QPX_ERR_INVALID_ARGUMENT.
qpx_next_deadline⚠
Writes the next timer deadline to out_ns, or returns QPX_EMPTY if none.
qpx_persisted⚠
Reports that every persist up to and including id is durable.
qpx_poll⚠
Takes the next action into out, or returns QPX_EMPTY.
qpx_propose⚠
Queues a write; its id is written to out_id.
qpx_propose_reconfiguration⚠
Proposes new membership: members_len ids at members, with order_len ids at leader_order as the preferred leaders (0 and null: by id). Its id is written to out_id and reported like a proposal’s; the change takes effect 64 slots after it is decided (QPX_ACTION_RECONFIGURED). QPX_ERR_RECONFIGURATION_PENDING while another change has not taken effect. A joining replica runs with the original configuration. To change only the leader order, pass the members in force and the new order. A change computed before another change or a rotation was decided is ignored and reported as QPX_ACTION_DROPPED with QPX_DROP_SUPERSEDED.
qpx_read_barrier⚠
Requests a linearizable read point; its id is written to out_id. Ids are opaque unsigned 64-bit values; read ids use the high bit (ABI 0.8).
qpx_replica_free⚠
Releases a handle (null is ignored).
qpx_replica_new⚠
Creates a replica with configuration cfg (copied; free it whenever) from n_records durable records (in persisted order).
qpx_status_str
Static, NUL-terminated description of a status code.
qpx_storage_failed⚠
Reports that persist id failed; the replica fences itself.