Module shared 

Shared SQLite storage for multiple workers.

This module provides SqliteStorageFactory, a factory for creating multiple SqliteStorage instances that share a single SQLite connection pool and task-fetching loop.

Why shared storage?

A SQLite backend can be used by multiple workers, but having every worker independently poll the database can result in unnecessary database queries and contention.

SqliteStorageFactory addresses this by maintaining a single shared polling task for all storage instances created by the factory.

When a task is inserted into the jobs table, SQLite’s update hook notifies the shared poller.

The poller fetches available tasks for all registered queues in a batch and routes each task to the SharedFetcher belonging to the corresponding storage instance.

The resulting architecture is roughly:

                        SQLite
                           │
                    update hook
                           │
                           ▼
                 ┌───────────────────┐
                 │  Shared poller    │
                 │                   │
                 │ fetches tasks for │
                 │ all registered    │
                 │ queues in batches │
                 └─────────┬─────────┘
                           │
             ┌─────────────┼─────────────┐
             │             │             │
             ▼             ▼             ▼
         queue A       queue B       queue C
             │             │             │
             ▼             ▼             ▼
         Worker A      Worker B      Worker C

Creating a factory

The factory owns the underlying SqlitePool and can create multiple backends from it:

let mut factory = SqliteStorageFactory::new("sqlite://jobs.db");

let first = factory.create().unwrap();
let second = factory.create().unwrap();

Each backend has its own queue registration and receiver, while the database connection pool and polling task are shared.

Queue registration

Each backend created by the factory is associated with a queue derived from its Config.

A queue may only be registered once with a factory.

Attempting to create another backend for an already-registered queue returns SharedSqliteError::NamespaceExists.

Task dispatch

SharedFetcher implements Stream and receives tasks from the shared polling loop through an asynchronous channel.

The shared poller keeps running while at least one fetcher is being polled and dispatches each fetched task to the channel associated with its queue.

The resulting backend is an Interleave combining the normal SqliteStorage implementation with SharedFetcher.

This allows task insertion and other storage operations to continue using the normal SQLite backend while task consumption is coordinated by the shared polling loop.

Pool configuration

SqliteStorageFactory::new creates a pool with unlimited connection lifetime and idle timeout. For applications requiring more control over the SQLite pool, SqliteStorageFactory::new_with_pool_options accepts custom PoolOptions.

Structs

SharedFetcher

A fetcher that uses a channel to receive jobs from a shared polling point

SqliteStorageFactory

Shared Sqlite storage backend that can be used across multiple workers

Enums

SharedSqliteError

Errors that can occur when creating a shared Sqlite storage backend

Type Aliases

SharedSqliteStorage

An SqliteStorage interleaving with SharedFetcher