Expand description
Stable C ABI over the quepaxa consensus core, for C, C++, Go, Swift, Kotlin/JNI,
Python (ctypes/cffi) or any language that can call C.
The ABI exposes the same sans-IO quepaxa::Replica: your program feeds it events and
carries out the actions it returns. Read quepaxa::guide::driving for the model; this
page covers what is specific to C.
§Getting the library
- Prebuilt: every GitHub release attaches
quepaxa-ffi-<version>-<target>archives withinclude/quepaxa.hand the static and shared libraries for Linux (x86_64, aarch64; glibc 2.28+), macOS (x86_64, arm64), Windows (x64, arm64, MSVC), Android (jniLibsfor four ABIs, 16 KB pages) and iOS (an XCFramework). - From source:
cargo build --release -p quepaxa-ffiproduceslibquepaxa_ffi.a/.so/.dylib(orquepaxa_ffi.lib/.dll) intarget/release; the header iscrates/quepaxa-ffi/include/quepaxa.h. - Linking statically on Linux also needs
-lpthread -ldl -lm.
§The loop in C
#include "quepaxa.h"
uint32_t members[3] = {0, 1, 2};
uint8_t seed[32];
fill_with_csprng(seed, sizeof seed); /* fresh on every start */
qpx_config *cfg;
check(qpx_config_new((const uint8_t *)"my-cluster-0001!", 0, members, 3, seed, &cfg));
check(qpx_config_set(cfg, QPX_CONFIG_MAX_IN_FLIGHT_SLOTS, 32)); /* optional settings */
qpx_replica *r;
check(qpx_replica_new(cfg, records, n_records, now_ns(), &r)); /* durable records, in order */
qpx_config_free(cfg); /* the replica keeps its own copy */
for (;;) { /* after every call: drain */
qpx_action a;
int32_t rc = qpx_poll(r, &a);
if (rc == QPX_EMPTY) break;
switch (a.kind) {
case QPX_ACTION_PERSIST: /* qpx_action_record(r, k, ...) for k < a.count: append + fsync, */
/* then send qpx_action_then(r, k, ...) for k < a.then_count, */
/* then, after draining, qpx_persisted(r, now, a.id) */
case QPX_ACTION_SEND: /* send a.bytes to a.peer */
case QPX_ACTION_COMMIT: /* apply qpx_action_entry(r, k, ...) for k < a.count, slot a.slot */
case QPX_ACTION_COMMITTED: /* answer the client of proposal a.id */
default: break; /* new kinds may appear in minor versions */
}
}
uint64_t deadline;
if (qpx_next_deadline(r, &deadline) == QPX_OK) { /* arm a timer; then qpx_handle_timeout */ }examples/smoke.c in the crate is a complete three-replica program; CI runs it on Linux,
macOS, Windows, an Android emulator and an iOS simulator.
§Contract
(Also stated in include/quepaxa.h.)
- Every function returns a
QPX_*status (0= ok, positive = informational, negative = error) unless stated otherwise.qpx_status_strnames it andqpx_last_errordescribes the most recent failing call on the calling thread. - Draining: after any mutating call, call
qpx_polluntil it returnsQPX_EMPTYbefore the next mutating call (QPX_ERR_NOT_DRAINEDotherwise).qpx_persistedis itself a mutating call: collect persist ids while draining and acknowledge the highest one afterwards (it covers every earlier persist). - Persist before send: never send a persist’s
thenmessages before its records are durable. - Borrowed memory:
qpx_bytesreturned byqpx_polland theqpx_action_*accessors borrow from the handle and stay valid until the next call on that handle; copy what you keep. Inputs are copied by the library; you keep ownership. - Threads: a handle may move between threads, but calls on one handle must not overlap. Different handles are independent.
- Panics never cross the boundary: a panic inside the library returns
QPX_ERR_PANICand poisons the handle; every later call on it returnsQPX_ERR_POISONED. Free it withqpx_replica_free.
§Versioning
qpx_abi_version returns (major << 16) | minor; compare it with
QPX_ABI_VERSION_MAJOR/MINOR from the header you compiled against. Settings and
metrics are not structs: the configuration is an opaque object set option by option
(qpx_config_set) and metrics are read one by one (qpx_metric), so a new one is a
new constant, and an option or metric the library does not know is an explicit
QPX_ERR_INVALID_ARGUMENT. The only structs are outputs with a fixed layout (qpx_bytes,
qpx_action, qpx_entry). Before 1.0 a minor ABI bump may be incompatible; the
changelog says when.
Structs§
- qpx_
action - One action. Borrowed fields stay valid until the next call on the handle.
- qpx_
bytes - A borrowed byte range.
ptrmay be null whenlenis 0. - qpx_
config - Replica configuration: create it with
qpx_config_new, adjust it withqpx_config_setandqpx_config_set_leader_order, pass it toqpx_replica_new(which copies it), and release it withqpx_config_free. - qpx_
entry - A committed entry.
- qpx_
replica - An opaque replica handle.
Constants§
- QPX_
ABI_ VERSION_ MAJOR - ABI major version: incompatible changes bump it. While it is 0 the ABI is unstable and every minor bump may be incompatible.
- QPX_
ABI_ VERSION_ MINOR - ABI minor version: additive changes bump it (any change while the major is 0).
0.2:
qpx_install_snapshotmay returnQPX_IGNORED. 0.3: size-awareqpx_config_init,qpx_get_metrics,qpx_last_error, strict draining. 0.4:qpx_config.leader_order(appended; older layouts keep the default order). 0.5: membership changes:qpx_propose_reconfiguration,QPX_ACTION_RECONFIGURED,qpx_action_member,QPX_ERR_RECONFIGURATION_PENDING. 0.6:QPX_ACTION_RETIRED. 0.7: opaqueqpx_config(qpx_config_new,qpx_config_set,qpx_config_set_leader_order,qpx_config_free) andqpx_metricreplace theqpx_configandqpx_metricsstructs,qpx_config_initandqpx_get_metrics(incompatible). 0.8:QPX_ERR_REQUEST_ID_EXHAUSTED; read ids use a separate range;QPX_METRIC_STEPPED_ASIDE. 0.9:QPX_DROP_SUPERSEDED,QPX_DROP_UNRESOLVED. 0.10: leader placement:QPX_CONFIG_PLACEMENT*,QPX_METRIC_SUSPICIONS,QPX_METRIC_PLACEMENT_MOVES,QPX_METRIC_EPOCH,QPX_METRIC_LEADER_POSITION. - QPX_
ACTION_ COMMIT - Apply the
countentries ofslot(read them withqpx_action_entry). - QPX_
ACTION_ COMMITTED - Local proposal
idcommitted at or beforeslot. - QPX_
ACTION_ DROPPED - Local proposal
idwill not be reported again (flags=QPX_DROP_*). WithQPX_DROP_FENCEDthe outcome is unknown: it may still commit, as after a crash. - QPX_
ACTION_ PERSIST - Append
recordsdurably after earlier persists, then send eachthenmessage, then callqpx_persisted(id). IfQPX_PERSIST_REPLACEis set inflags, atomically replace all earlier records instead of appending. - QPX_
ACTION_ READ_ READY - Read barrier
idis ready once commits up toslotare applied. - QPX_
ACTION_ RECONFIGURED - A membership change was applied: from slot
slot, thecountmembers (read them withqpx_action_member, preferred leaders first) form epochid(ABI 0.5). - QPX_
ACTION_ RETIRED - This replica is not a member of epoch
id, and a majority of that epoch’s members durably know every slot before it: it may shut down, unless a later change added it back (ABI 0.6). - QPX_
ACTION_ SEND - Send
bytestopeer. - QPX_
ACTION_ SNAPSHOT_ NEEDED - Fetch
peer’s application snapshot coveringslotand callqpx_install_snapshot. - QPX_
BUFFER_ TOO_ SMALL - The output buffer is too small; the required length was written.
- QPX_
CONFIG_ HEDGE_ MAX_ NS - Upper bound of the hedge delay, nanoseconds.
- QPX_
CONFIG_ HEDGE_ MIN_ NS - Lower bound of the hedge delay, nanoseconds.
- QPX_
CONFIG_ MAX_ BATCH_ BYTES - Largest batch of entries per slot, bytes (
u32). - QPX_
CONFIG_ MAX_ IN_ FLIGHT_ SLOTS - Concurrent slots driven by this replica (
u32, at most 64). - QPX_
CONFIG_ MAX_ MESSAGE_ BYTES - Largest accepted message, bytes (
u32). - QPX_
CONFIG_ MAX_ OUTSTANDING_ PROPOSALS - Outstanding proposals before
QPX_ERR_BUSY(u32). - QPX_
CONFIG_ PLACEMENT - Leader placement on this member: 1 on, 0 off (the default). Enable it on every member (ABI 0.10).
- QPX_
CONFIG_ PLACEMENT_ HOLD_ NS - How long placement leaves any change of the leader order alone, nanoseconds (ABI 0.10).
- QPX_
CONFIG_ PLACEMENT_ SUSTAIN_ NS - How long a member must keep a 20% gain before placement moves the leader to it, nanoseconds (ABI 0.10).
- QPX_
CONFIG_ RETRANSMIT_ INTERVAL_ NS - Record request retransmission interval, nanoseconds.
- QPX_
DROP_ FENCED flagsofQPX_ACTION_DROPPED: the replica is fenced (outcome unknown).- QPX_
DROP_ SUPERSEDED flagsofQPX_ACTION_DROPPED: a membership change was decided but ignored, because another change or a rotation came first. Nothing changed (ABI 0.9).- QPX_
DROP_ UNRESOLVED flagsofQPX_ACTION_DROPPED: a membership change was decided, but an installed snapshot hid whether it was applied; check the members in force (ABI 0.9).- QPX_
EMPTY - No action, deadline or value to report.
- QPX_
ERR_ BUSY - Too many outstanding proposals.
- QPX_
ERR_ CORRUPT_ RECORD - A durable record passed to
qpx_replica_newis corrupt. - QPX_
ERR_ FENCED - The replica fenced itself after a storage failure.
- QPX_
ERR_ INVALID_ ARGUMENT - A null pointer, unknown option or metric, out-of-range value or index.
- QPX_
ERR_ INVALID_ CONFIG - The configuration is invalid.
- QPX_
ERR_ INVALID_ CORE_ STATE - Malformed core state for
qpx_compact/qpx_install_snapshot. - QPX_
ERR_ NOT_ DRAINED - Actions from the previous call were not drained.
- QPX_
ERR_ PANIC - The library panicked; the handle is now poisoned.
- QPX_
ERR_ PAYLOAD_ TOO_ LARGE - The payload does not fit in one batch.
- QPX_
ERR_ POISONED - The handle panicked earlier and is unusable.
- QPX_
ERR_ RECONFIGURATION_ PENDING - A membership change has not taken effect yet (ABI 0.5).
- QPX_
ERR_ REQUEST_ ID_ EXHAUSTED - Request ids exhausted in this incarnation; restart from durable records (ABI 0.8).
- QPX_
ERR_ UNKNOWN_ PERSIST_ ID - Unknown or repeated persist id.
- QPX_
IGNORED - The call had no effect: the snapshot is older than what the replica already emitted.
- QPX_
METRIC_ CATCH_ UP_ RETRIES - Catch-up requests re-sent.
- QPX_
METRIC_ CORRUPT_ BATCHES - Decided batches that failed to decode.
- QPX_
METRIC_ EPOCH - Gauge: membership epoch in force (ABI 0.10).
- QPX_
METRIC_ FAST_ PATH_ DECISIONS - Own decisions on the leader fast path.
- QPX_
METRIC_ FORWARDED - Batches forwarded to the leader.
- QPX_
METRIC_ HEDGED - Own entries proposed directly after the leader was too slow.
- QPX_
METRIC_ LEADER_ POSITION - Gauge: this replica’s position in the leader order in force, 0 when it leads (ABI 0.10).
- QPX_
METRIC_ LIVE_ REGISTERS - Gauge: recorder registers held in memory.
- QPX_
METRIC_ MESSAGES_ REJECTED - Network messages rejected (malformed, foreign or from non-members).
- QPX_
METRIC_ PLACEMENT_ MOVES - Leader-order changes leader placement proposed from this replica (ABI 0.10).
- QPX_
METRIC_ RETAINED_ SLOTS - Gauge: decided slots held in memory.
- QPX_
METRIC_ RETRANSMITS - Record requests re-sent.
- QPX_
METRIC_ SLOTS_ DECIDED - Slots learned as decided.
- QPX_
METRIC_ STEPPED_ ASIDE - Times this replica stepped aside because a write waited too long on its disk.
- QPX_
METRIC_ SUSPICIONS - Times this replica suspected the member it hands entries to of being silent (ABI 0.10).
- QPX_OK
- Success.
- QPX_
PERSIST_ REPLACE flagsbit ofQPX_ACTION_PERSIST: replace all earlier records.
Functions§
- qpx_
abi_ version (major << 16) | minor.- qpx_
action_ ⚠entry - Entry
iof the current commit action. - qpx_
action_ ⚠member - Member
iof the currentQPX_ACTION_RECONFIGUREDaction (preferred leaders first). - qpx_
action_ ⚠record - Record
iof the current persist action. - qpx_
action_ ⚠then - Message
igated on the current persist action, and its destination. - qpx_
commit_ ⚠index - Number of contiguous decided slots.
- qpx_
compact ⚠ - Discards log state up to
through, snapshotted by the application withstate. - qpx_
config_ ⚠free - Releases a configuration (null is ignored). Replicas created from it are unaffected.
- qpx_
config_ ⚠new - Creates a configuration with default settings:
cluster_id(16 bytes; messages from other clusters are dropped), this replica’sid, themembers_lenconfigured members atmembers(including this one; a joining replica lists the original members), andseed(32 bytes of fresh CSPRNG output on every start; never reuse it). - qpx_
config_ ⚠set - Sets option
option(QPX_CONFIG_*) tovalue. An unknown option, or a value too large for it, isQPX_ERR_INVALID_ARGUMENT; whether the settings are consistent is checked byqpx_replica_new(QPX_ERR_INVALID_CONFIG). - qpx_
config_ ⚠set_ leader_ order - Sets the preferred leaders, first to last: every member once, or none (
len0) for “by id”. Every member must use the same order. - qpx_
core_ ⚠state - Copies the replica’s core state into
buf. Two-call pattern: with too small acapit returnsQPX_BUFFER_TOO_SMALLafter writing the needed size toout_len. - qpx_
handle_ ⚠message - Feeds a network message exactly as received.
- qpx_
handle_ ⚠timeout - Fires timers due at
now_ns. - qpx_
install_ ⚠snapshot - Adopts a snapshot covering slots up to
through;QPX_IGNOREDif the replica already emitted past it (do not rewind the application state in that case). - qpx_
last_ ⚠error - Copies the description of the calling thread’s most recent failing call (empty if the
last call succeeded) as a NUL-terminated string. Two-call pattern: with too small a
capit returnsQPX_BUFFER_TOO_SMALLafter writing the needed size (including the NUL) toout_len. - qpx_
metric ⚠ - Reads counter or gauge
metric(QPX_METRIC_*, see the RustMetrics) intoout; an unknown metric isQPX_ERR_INVALID_ARGUMENT. - qpx_
next_ ⚠deadline - Writes the next timer deadline to
out_ns, or returnsQPX_EMPTYif none. - qpx_
persisted ⚠ - Reports that every persist up to and including
idis durable. - qpx_
poll ⚠ - Takes the next action into
out, or returnsQPX_EMPTY. - qpx_
propose ⚠ - Queues a write; its id is written to
out_id. - qpx_
propose_ ⚠reconfiguration - Proposes new membership:
members_lenids atmembers, withorder_lenids atleader_orderas the preferred leaders (0 and null: by id). Its id is written toout_idand reported like a proposal’s; the change takes effect 64 slots after it is decided (QPX_ACTION_RECONFIGURED).QPX_ERR_RECONFIGURATION_PENDINGwhile another change has not taken effect. A joining replica runs with the original configuration. To change only the leader order, pass the members in force and the new order. A change computed before another change or a rotation was decided is ignored and reported asQPX_ACTION_DROPPEDwithQPX_DROP_SUPERSEDED. - qpx_
read_ ⚠barrier - Requests a linearizable read point; its id is written to
out_id. Ids are opaque unsigned 64-bit values; read ids use the high bit (ABI 0.8). - qpx_
replica_ ⚠free - Releases a handle (null is ignored).
- qpx_
replica_ ⚠new - Creates a replica with configuration
cfg(copied; free it whenever) fromn_recordsdurable records (in persisted order). - qpx_
status_ str - Static, NUL-terminated description of a status code.
- qpx_
storage_ ⚠failed - Reports that persist
idfailed; the replica fences itself.