Struct ocypod::application::RedisManager

pub struct RedisManager {

    pub queues_key: String,
    pub limbo_key: String,
    pub running_key: String,
    pub failed_key: String,
    pub ended_key: String,
    pub job_id_key: String,
    pub queue_prefix: String,
    pub job_prefix: String,
    pub tag_prefix: String,
    pub stat_jobs_created_key: String,
    pub stat_jobs_completed_key: String,
    pub stat_jobs_retried_key: String,
    pub stat_jobs_failed_key: String,
    pub stat_jobs_timed_out_key: String,
    pub stat_jobs_cancelled_key: String,
}

Manages queues and jobs within Redis. Contains main public functions that are called by HTTP services.

Internally, uses RedisJob and RedisQueue structs as convenient wrappers around interacting with jobs/queues.

Fields

queues_key: String

Redis key for list of all queues. This is used for fast lookups of all queue names without a scan.

limbo_key: String

Redis key for limbo queue. This is a very short lived queue, used to keep jobs in the transition state between queued and running. It’s mostly a workaround for not being able to atomically pop a job from a queue and update its metadata (stored in a separate hash) without the risk of losing some data.

running_key: String

Redis key for the running job list. Jobs are moved here from their original queue (via limbo) when they’re picked up by a worker. Jobs in this queue are checked for timeouts.

failed_key: String

Redis key for the failed job list. Jobs that have either timed out, or failed by worker request are moved to this queue. Jobs in this queue are monitored for retries.

ended_key: String

Redis key for the ended job list. Jobs are moved here then they have either successfully completed, or failed/timed out with no remaining retries to attempted. Jobs in this queue are monitored for expiry.

job_id_key: String

Redis key for the job ID counter. This is used as a counter to generate unique IDs for each job.

queue_prefix: String

Prefix used for queue settings keys in Redis. A user created queue with name “foo” have its configuration stored under the key “queue:foo”.

job_prefix: String

Prefix used for job keys in Redis. A job with the ID 123 would be stored under the key “job:123”.

tag_prefix: String

Prefix used for tag keys in Redis. These are used to index jobs by any tags they were given at creation time. A tag created with name “foo” would be stored a “tag:foo”.

stat_jobs_created_key: String

Prefix used for job created statistics.

stat_jobs_completed_key: String

Prefix used for job completed statistics.

stat_jobs_retried_key: String

Prefix used for job retry statistics.

stat_jobs_failed_key: String

Prefix used for job failed statistics.

stat_jobs_timed_out_key: String

Prefix used for job timed out statistics.

stat_jobs_cancelled_key: String

Prefix used for job cancelled statistics.

Implementations

impl RedisManager

pub fn new(key_namespace: &str) -> Self

Creates a new RedisManager which uses the given namespace prefix for internal keys it uses. If the given namespace is empty, then no prefix is used.

pub async fn create_or_update_queue<C: ConnectionLike>(
    &self,
    conn: &mut C,
    name: &str,
    settings: &Settings
) -> OcyResult<bool>

Create or update a queue in Redis with given name and settings.

Returns true if a new queue was created, or false if an existing queue was updated.

pub async fn delete_queue<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    name: &str
) -> OcyResult<bool>

Delete queue with given name from Redis.

Returns true if a queue was deleted, and false if no queue with given name was found.

pub async fn delete_job<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    job_id: u64
) -> OcyResult<bool>

Delete a job with given ID from Redis.

Returns true if a job was found and deleted, false if no job with given ID was found.

pub async fn server_info<C: ConnectionLike + Send>(
    &self,
    conn: &mut C
) -> OcyResult<ServerInfo>

Get summary of server and queue data. Currently contains:

• count of each job’s status by queue
• total number of jobs processed and their final status

pub async fn job_fields<C: ConnectionLike>(
    &self,
    conn: &mut C,
    job_id: u64,
    fields: Option<&[Field]>
) -> OcyResult<JobMeta>

Get one or more metadata fields from given job ID.

If None is given as the fields argument, then get all fields.

pub async fn update_job<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    job_id: u64,
    update_req: &UpdateRequest
) -> OcyResult<()>

Update one or more job metadata fields.

Only following fields can be updated in this way:

• status - used to mark job as completed/failed/cancelled etc.
• output - used to update user provided information related to this job

pub async fn update_job_heartbeat<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    job_id: u64
) -> OcyResult<()>

Update a job’s last_heartbeat field with the current date/time.

pub async fn job_status<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    job_id: u64
) -> OcyResult<Status>

Get the status field of given job.

pub async fn set_job_status<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    job_id: u64,
    status: &Status
) -> OcyResult<()>

Update a job’s status field to the given status, if an allowed state transition.

Identical to calling update_job and with Some(status) provided.

pub async fn job_output<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    job_id: u64
) -> OcyResult<Value>

Get the output field of given job.

pub async fn set_job_output<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    job_id: u64,
    value: &Value
) -> OcyResult<()>

Update a job’s output field to the given output data.

Identical to calling update_job and with Some(output) provided.

pub async fn tagged_job_ids<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    tag: &str
) -> OcyResult<Vec<u64>>

Get a list of jobs IDs with given tag name.

pub async fn queue_names<C: ConnectionLike + Send>(
    &self,
    conn: &mut C
) -> OcyResult<Vec<String>>

Get list of all queue names.

pub async fn queue_settings<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    queue_name: &str
) -> OcyResult<Settings>

Get given queue’s current settings.

pub async fn queue_size<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    queue_name: &str
) -> OcyResult<u64>

Get the number of queues jobs in given queue.

pub async fn running_queue_size<C: ConnectionLike + Send>(
    &self,
    conn: &mut C
) -> OcyResult<u64>

Get total number of running jobs across all queues.

pub async fn failed_queue_size<C: ConnectionLike + Send>(
    &self,
    conn: &mut C
) -> OcyResult<u64>

Get total number of failed jobs across all queues.

pub async fn ended_queue_size<C: ConnectionLike + Send>(
    &self,
    conn: &mut C
) -> OcyResult<u64>

Get total number of ended jobs across all queues.

pub async fn queue_job_ids<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    queue_name: &str
) -> OcyResult<HashMap<Status, Vec<u64>>>

Get a list of job IDs that are currently in a given queue.

pub async fn check_job_retries<C: ConnectionLike + Send>(
    &self,
    conn: &mut C
) -> OcyResult<Vec<u64>>

Check all jobs in the failed queue for retries.

Any which can be retried are re-queued on the queue they were created it.

Any which have no automatic retries remaining are moved to the ended queue.

pub async fn check_job_timeouts<C: ConnectionLike + Send>(
    &self,
    conn: &mut C
) -> OcyResult<Vec<u64>>

Check all jobs in the running queue for timeouts.

Any which timeout are moved to the failed queue, where they’ll eventually either be retried, or moved to the ended queue.

pub async fn check_job_expiry<C: ConnectionLike + Send>(
    &self,
    conn: &mut C
) -> OcyResult<Vec<u64>>

Check all jobs in the ended queue for expiry. Any expired jobs will be entirely removed from the queue system.

pub async fn check_db_integrity<C: ConnectionLike + Send>(
    &self,
    conn: &mut C
) -> OcyResult<()>

Checks the integrity of Redis DB, e.g. checking for dangling indexes, jobs in invalid states, etc.

Mostly intended for use during development, as it has a non-trivial runtime cost.

pub async fn check_ping<C: ConnectionLike>(conn: &mut C) -> OcyResult<()>

Check connection to Redis using ping command.

pub async fn next_queued_job<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    queue_name: &str
) -> OcyResult<Option<Payload>>

Fetch the next job from given queue, if any.

Returns

A job::Payload if a job is found, or None if the queue is empty.

pub async fn create_job<C: ConnectionLike + Send>(
    &self,
    conn: &mut C,
    queue_name: &str,
    job_req: &CreateRequest
) -> OcyResult<u64>

Create a new job on given queue.

pub fn build_tag_key(&self, tag: &str) -> OcyResult<String>

Get unique Redis key for given tag.

Trait Implementations

impl Clone for RedisManager

fn clone(&self) -> RedisManager

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RedisManager

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.