Crate opsqueue

Opsqueue: A lightweight batch processing queue for heavy loads

Simple ‘getting started’ instructions can be found in the repository README.

The Rust codebase defines both the ‘server’, which makes up the bulk of the Opsqueue binary itself, and the ‘client’, which is the common part of functionality that clients written in different other programming languages can all use.

Many datatypes are shared between this server and client, and therefore their code lives together in the same crate. Instead, we use feature-flags (client-logic and server-logic) to decide what concrete parts to include when building. The set of dependencies is based on these same feature-flags. Most interestingly, in the test suite we enable both feature-flags so we’re able to do a bunch of round-trip testing immediately in Rust code.

Module setup

• The basic logic is divided in the producer and consumer modules. These both have their own db submodule.
• Common functionality and datatypes exists in the common module
• Common database helpers live in the db module.
• Reading/writing to object stores like GCS or S3 is abstracted in the object_store module.
• Finally, extra modules to have a single source of truth for configuration of the queue, and to nicely do tracing and expose metrics exist.

Modules

common

Common datatypes and errors shared across all parts of Opsqueue

config

Defines the source of truth for configuring the Opsqueue queue

consumer

The Consumer side: Interface to reserve, work on, and complete/fail individual Chunks

db

Core database abstractions for the opsqueue server internals.

producer

The Producer side: Interface to create and read submissions

prometheus

Define common metrics exposed via a Prometheus endpoint

server

Defines the HTTP endpoints that are used by both the producer and consumer APIs

tracing

Helpers to read/write OpenTelemetry Tracing contexts from inside submissions stored in the queue

Macros

E

Builds a nested E from two or more (error) types.

fold_both

Similar to map_both but doesn’t wrap the result back in the respective Left/Right variant.

map_both

Allows you to run the same expression on both halves of an E, without the types necessarily having to match.

Constants

VERSION_CARGO_SEMVER

The Opsqueue library’s semantic version as written in the Rust packages’s Cargo.toml

Functions

version_info