URI Register

Beta Software: This library is in active development and the API may change. While it's being used in production environments, you should pin to a specific version and test thoroughly before upgrading.

A caching, async-first PostgreSQL-backed URI register service for assigning unique integer IDs to URIs. Perfect for string interning, deduplication, and systems that need consistent global identifier mappings.

Note: This library is async-only and requires an async runtime (tokio).

Overview

The URI Register provides a simple, fast way to assign unique integer IDs to URI strings. Once registered, a URI always returns the same ID, making it ideal for string interning and deduplication in distributed systems.

Features

• Simple API: Just 2 methods - register_uri() and register_uri_batch()
• Async-only: Built on tokio/sqlx for high concurrency
• Batch optimised: Process thousands of URIs in a single database round-trip
• Configurable caching: W-TinyLFU (Moka) or LRU caching for frequently accessed URIs
• Order preservation: Batch operations maintain strict order correspondence
• PostgreSQL backend: Durable, scalable, with connection pooling
• Automatic retry logic: Configurable exponential backoff for transient database errors
• Thread-safe: Designed for concurrent access from multiple threads/processes

Use Cases

• String interning systems: Reduce memory footprint by storing strings once and referencing by ID
• URL deduplication: Assign unique IDs to URLs across distributed crawlers
• Global identifier systems: Centralised ID assignment for URIs/strings in microservices
• Data warehousing: Efficient storage of repeated string values
• Distributed caching: Consistent ID assignment across cache nodes

Installation

Rust

Add to your Cargo.toml:

[dependencies]
uri-register = "0.1.2"

Or use as a git dependency:

[dependencies]
uri-register = { git = "https://github.com/telicent-oss/uri-register" }

Python

Install from TestPyPI (during beta):

pip install --index-url https://test.pypi.org/simple/ uri-register

Requirements: Python 3.8+

Note: The package is currently published to TestPyPI for testing. Once stable, it will be available on the main PyPI repository.

Setup

1. Database Initialisation

Before using the URI Register service, you must initialise the PostgreSQL schema.

Run the schema creation script:

psql -U username -d database_name -f schema.sql

Or execute the SQL directly:

CREATE TABLE IF NOT EXISTS uri_register (
    id BIGSERIAL PRIMARY KEY,
    uri TEXT NOT NULL UNIQUE
);

2. Database Configuration

The service requires a PostgreSQL connection string. Set it as an environment variable or pass it directly:

export DATABASE_URL="postgresql://username:password@localhost:5432/database_name"

Usage

Rust Example

use uri_register::{CacheStrategy, PostgresUriRegister, UriService};

#[tokio::main]
async fn main() -> uri_register::Result<()> {
    // Connect to PostgreSQL
    let register = PostgresUriRegister::new(
        "postgres://localhost/mydb",
        "uri_register",       // table name
        20,                   // max connections
        10_000,               // cache size
        CacheStrategy::Moka,  // cache strategy (Moka is default, or use CacheStrategy::Lru)
        3,                    // max retries
        100,                  // initial backoff (ms)
        5000                  // max backoff (ms)
    ).await?;

    // Register a single URI
    let id = register.register_uri("http://example.org/resource/1").await?;
    println!("Registered URI with ID: {}", id);

    // Register the same URI again - returns the same ID
    let same_id = register.register_uri("http://example.org/resource/1").await?;
    assert_eq!(id, same_id);

    // Register multiple URIs in batch (much faster!)
    let uris = vec![
        "http://example.org/resource/2".to_string(),
        "http://example.org/resource/3".to_string(),
        "http://example.org/resource/4".to_string(),
    ];
    let ids = register.register_uri_batch(&uris).await?;

    // IDs maintain order: ids[i] corresponds to uris[i]
    for (uri, id) in uris.iter().zip(ids.iter()) {
        println!("{} -> {}", uri, id);
    }

    Ok(())
}

Python Example

import asyncio
from uri_register import UriRegister

async def main():
    # Connect to PostgreSQL
    register = await UriRegister.new(
        "postgres://localhost/mydb",
        "uri_register",  # table name
        20,              # max connections
        10_000,          # cache size
        "moka",          # cache strategy ("moka" is default, or use "lru")
        3,               # max retries
        100,             # initial backoff (ms)
        5000             # max backoff (ms)
    )

    # Register a single URI
    id = await register.register_uri("http://example.org/resource/1")
    print(f"Registered URI with ID: {id}")

    # Register the same URI again - returns the same ID
    same_id = await register.register_uri("http://example.org/resource/1")
    assert id == same_id

    # Register multiple URIs in batch (much faster!)
    uris = [
        "http://example.org/resource/2",
        "http://example.org/resource/3",
        "http://example.org/resource/4",
    ]
    ids = await register.register_uri_batch(uris)

    # IDs maintain order: ids[i] corresponds to uris[i]
    for uri, id in zip(uris, ids):
        print(f"{uri} -> {id}")

    # Get statistics
    stats = await register.stats()
    print(f"Total URIs: {stats['total_uris']}")

asyncio.run(main())

API Reference

The UriService trait provides two methods:

register_uri(uri: &str) -> u64

Register a single URI and return its ID.

• If the URI exists, returns the existing ID
• If the URI is new, creates a new ID and returns it
• Uses configurable cache (Moka/LRU) for fast repeated lookups

let id = register.register_uri("http://example.org/page").await?;

register_uri_batch(uris: &[String]) -> Vec<u64>

Register multiple URIs in batch and return their IDs.

• Order preserved: ids[i] corresponds to uris[i]
• Much faster than calling register_uri() multiple times
• Handles duplicate URIs in input correctly
• Cache-optimised: only queries database for cache misses

let uris = vec![
    "http://example.org/page1".to_string(),
    "http://example.org/page2".to_string(),
];
let ids = register.register_uri_batch(&uris).await?;

// Access by index
assert_eq!(ids[0], register.register_uri("http://example.org/page1").await?);

Statistics

Get information about the register:

let stats = register.stats().await?;
println!("Total URIs: {}", stats.total_uris);
println!("Storage size: {} bytes", stats.size_bytes);

Retry Configuration and Resilience

The URI Register includes automatic retry logic with exponential backoff to handle transient database errors. This improves reliability in production environments where temporary network issues or database load spikes may occur.

Retry Behavior

The service automatically retries operations that fail due to transient errors, including:

• Connection timeouts and connection pool exhaustion
• Network connectivity issues
• Database temporarily unavailable
• Deadlock errors and serialization failures
• Other transient PostgreSQL errors

Non-transient errors (such as constraint violations, invalid SQL, or configuration errors) are not retried and return immediately.

Configuration Parameters

When creating a PostgresUriRegister instance, three retry parameters are required:

• max_retries: Maximum number of retry attempts (recommended: 3, set to 0 to disable retries)
• initial_backoff_ms: Initial backoff delay in milliseconds (recommended: 100-500)
• max_backoff_ms: Maximum backoff delay in milliseconds (recommended: 5000-10000)

Retry Algorithm

The retry mechanism uses exponential backoff with jitter:

1. On first failure, wait initial_backoff_ms ± 25% random jitter
2. On subsequent failures, double the backoff delay (capped at max_backoff_ms)
3. Random jitter (±25%) prevents thundering herd problems
4. After max_retries attempts, return the error to the caller

Examples

Standard configuration (3 retries, 100ms-5s backoff):

let register = PostgresUriRegister::new(
    db_url,
    "uri_register",
    20,
    10_000,
    CacheStrategy::Moka,  // cache strategy
    3,                    // retry up to 3 times
    100,                  // start with 100ms delay
    5000                  // cap at 5s delay
).await?;

Aggressive retries (10 retries, 50ms-30s backoff):

let register = PostgresUriRegister::new(
    db_url,
    "uri_register",
    20,
    10_000,
    CacheStrategy::Moka,  // cache strategy
    10,                   // retry up to 10 times
    50,                   // start with 50ms delay
    30000                 // cap at 30s delay
).await?;

Disable retries (for testing or when application handles retries):

let register = PostgresUriRegister::new(
    db_url,
    "uri_register",
    20,
    10_000,
    CacheStrategy::Moka,  // cache strategy
    0,                    // no retries
    100,
    1000
).await?;

Cache Strategies

The URI register supports two caching strategies:

Moka (W-TinyLFU) - Default

Recommended for most workloads. W-TinyLFU (Window Tiny Least Frequently Used) combines recency and frequency tracking to provide better cache hit rates than plain LRU, especially for workloads with mixed hot/cold data.

use uri_register::CacheStrategy;

let register = PostgresUriRegister::new(
    db_url,
    "uri_register",
    20,
    10_000,
    CacheStrategy::Moka,  // W-TinyLFU algorithm
    3,
    100,
    5000
).await?;

Python:

register = await UriRegister.new(
    db_url,
    "uri_register",
    20,
    10_000,
    "moka",  # W-TinyLFU algorithm
    3,
    100,
    5000
)

LRU (Least Recently Used)

Simple eviction based on recency of access. Use this if you have specific requirements or want more predictable behavior.

let register = PostgresUriRegister::new(
    db_url,
    "uri_register",
    20,
    10_000,
    CacheStrategy::Lru,  // Simple LRU
    3,
    100,
    5000
).await?;

Python:

register = await UriRegister.new(
    db_url,
    "uri_register",
    20,
    10_000,
    "lru",  # Simple LRU
    3,
    100,
    5000
)

Performance Comparison:

For most real-world workloads, Moka (W-TinyLFU) provides 10-30% better cache hit rates compared to LRU, especially when:

• Access patterns have varying frequency (some URIs accessed much more than others)
• There are periodic "scans" or one-time accesses that would pollute an LRU cache
• Working set size is close to cache capacity

Performance

Logged Tables (Default)

With default logged tables on typical hardware:

• Single registration: ~500-1K URIs/sec (with cache: 100K+/sec)
• Batch registration: ~10K-50K URIs/sec
• Batch lookup (cached): ~1M+ URIs/sec (no DB round-trip)
• Batch lookup (uncached): ~100K-200K URIs/sec

Unlogged Tables (Optional)

For 2-3x faster writes at the cost of durability:

ALTER TABLE uri_register SET UNLOGGED;

Performance with unlogged tables:

• Batch registration: ~30K-150K URIs/sec

WARNING: Unlogged tables lose all data if PostgreSQL crashes. Only use this if you can rebuild the register from source data.

To revert back to logged mode:

ALTER TABLE uri_register SET LOGGED;

Performance Tips

1. Always use batch operations when processing multiple URIs
2. Configure connection pooling appropriately for your workload (typical: 10-50 connections)
3. Tune cache size based on your working set size and available memory (typical: 10,000-100,000 entries)
4. Batch size: Optimal batch size is typically 1,000-10,000 URIs per operation
5. Indexing: The URI index is essential for lookup performance
6. Consider unlogged tables for initial bulk loading, then switch to logged

Architecture

Application
    ↓
UriService trait (2 methods)
    ↓
PostgresUriRegister impl
    ↓  ↓
Cache (Moka/LRU)  Connection Pool (20 connections)
    ↓                   ↓
    └───────────────→ PostgreSQL Database

Schema Details

The register uses a simple two-column table:

• id: BIGSERIAL primary key (auto-incrementing u64)
• uri: TEXT with UNIQUE constraint (indexed)

The UNIQUE constraint prevents duplicate URIs, and the index provides fast lookups.

Testing

For testing purposes, an in-memory implementation is available:

#[cfg(test)]
use uri_register::InMemoryUriRegister;

#[tokio::test]
async fn test_uri_register() {
    let register = InMemoryUriRegister::new();
    let id = register.register_uri("http://example.org").await.unwrap();
    assert_eq!(id, 1); // First URI gets ID 1
}

Error Handling

The library uses structured error types for better error handling and programmatic error inspection:

use uri_register::{CacheStrategy, ConfigurationError, Error, Result};

// Configuration errors with specific variants
match PostgresUriRegister::new("postgres://localhost/db", "uri_register", 0, 10_000, CacheStrategy::Moka, 3, 100, 5000).await {
    Ok(register) => { /* use register */ },
    Err(Error::Configuration(ConfigurationError::InvalidMaxConnections(n))) => {
        eprintln!("Invalid max_connections: {}", n);
    },
    Err(Error::Configuration(ConfigurationError::InvalidCacheSize(n))) => {
        eprintln!("Invalid cache_size: {}", n);
    },
    Err(Error::Configuration(ConfigurationError::InvalidTableName(msg))) => {
        eprintln!("Invalid table_name: {}", msg);
    },
    Err(Error::Configuration(ConfigurationError::InvalidBackoff(msg))) => {
        eprintln!("Invalid backoff configuration: {}", msg);
    },
    Err(e) => eprintln!("Other error: {}", e),
}

// Database errors (connection strings are sanitised to prevent password leaks)
match register.register_uri("http://example.org").await {
    Ok(id) => println!("Registered with ID: {}", id),
    Err(Error::Database(msg)) => eprintln!("Database error: {}", msg),
    Err(Error::InvalidUri(msg)) => eprintln!("Invalid URI: {}", msg),
    Err(e) => eprintln!("Other error: {}", e),
}

Error Types

• Configuration - Invalid configuration parameters (structured with specific variants)
• Database - Database operation failures (error messages sanitised)
• ConnectionPool - Connection pool errors
• Cache - Cache operation failures
• InvalidUri - URI validation failures (non-RFC 3986 compliant URIs)

License

Licensed under the Apache License, Version 2.0 (LICENSE or http://www.apache.org/licenses/LICENSE-2.0).

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.