json-register 0.1.2

A Rust library for registering JSON objects in PostgreSQL with canonicalisation and caching
Documentation

json-register

CI

Note: This library is currently in beta. The API is stable but may change in future releases based on user feedback and production usage.

json-register is a caching registry for JSON objects, with storage in a PostgreSQL database, using their JSONB encoding. It ensures that semantically equivalent JSON objects are cached only once by employing a canonicalisation strategy in the cache, and using JSONB comparisons in the database. The database assigns a uniqiue 32-bit integer identifier to each object.

This library is written in Rust and provides native bindings for Python, allowing for seamless integration into applications written in either language.

Features

  • Canonicalisation: JSON objects are canonicalised (keys sorted, whitespace removed) before storage to ensure uniqueness based on content.
  • Caching: An in-memory Least Recently Used (LRU) cache minimizes database lookups for frequently accessed objects.
  • PostgreSQL Integration: Efficiently stores and retrieves JSON data using PostgreSQL's JSONB type.
  • Batch Processing: Supports batch registration of objects to reduce network round-trips and improve throughput.
  • Cross-Language Support: Provides a native Rust API and a Python extension module.
  • Security: SQL injection prevention through identifier validation and automatic password sanitization in error messages.
  • Configurable Timeouts: Optional connection pool timeouts for acquire, idle, and maximum lifetime settings.
  • Monitoring: Query methods for connection pool metrics and cache hit rate statistics.

Installation

Rust

Add the following to your Cargo.toml:

[dependencies]
json-register = "0.1.0"
tokio = { version = "1.0", features = ["full"] }
serde_json = "1.0"

Python

Ensure you have a compatible Python environment (3.8+) and install the package.

Currently available on TestPyPI:

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ json-register-rust

Once published to PyPI:

pip install json-register-rust

Usage

Rust Example

The following example demonstrates how to initialize the registry and register JSON objects using the Rust API.

use json_register::Register;
use serde_json::json;
use std::error::Error;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // Configuration parameters
    let connection_string = "postgres://user:password@localhost:5432/dbname";
    let table_name = "json_objects";
    let id_column = "id";
    let jsonb_column = "data";
    let pool_size = 10;
    let lru_cache_size = 1000;

    // Initialize the register
    let register = Register::new(
        connection_string,
        table_name,
        id_column,
        jsonb_column,
        pool_size,
        lru_cache_size,
        None, // acquire_timeout_secs (defaults to 5)
        None, // idle_timeout_secs (defaults to 600)
        None, // max_lifetime_secs (defaults to 1800)
    ).await?;

    // Register a single object
    let object = json!({
        "name": "Alice",
        "role": "Engineer",
        "active": true
    });

    let id = register.register_object(&object).await?;
    println!("Registered object with ID: {}", id);

    // Register a batch of objects
    let batch = vec![
        json!({"name": "Bob", "role": "Manager"}),
        json!({"name": "Charlie", "role": "Designer"}),
    ];

    let ids = register.register_batch_objects(&batch).await?;
    println!("Registered batch IDs: {:?}", ids);

    Ok(())
}

Python Example

The following example demonstrates how to use the library within a Python application.

from json_register import JsonRegister
import asyncio

def main():
    # Initialize the register
    # Note: The Python constructor accepts individual connection parameters.
    register = JsonRegister(
        database_name="dbname",
        database_host="localhost",
        database_port=5432,
        database_user="user",
        database_password="password",
        lru_cache_size=1000,
        table_name="json_objects",
        id_column="id",
        jsonb_column="data",
        pool_size=10
    )

    # Register a single object
    obj = {
        "name": "Alice",
        "role": "Engineer",
        "active": True
    }
    
    # The register_object method is synchronous in the Python bindings
    # as it handles the async runtime internally.
    obj_id = register.register_object(obj)
    print(f"Registered object with ID: {obj_id}")

    # Register a batch of objects
    batch = [
        {"name": "Bob", "role": "Manager"},
        {"name": "Charlie", "role": "Designer"}
    ]
    
    batch_ids = register.register_batch_objects(batch)
    print(f"Registered batch IDs: {batch_ids}")

if __name__ == "__main__":
    main()

Configuration

Timeout Parameters

Optional timeout parameters can be specified when initializing the register. All timeouts are in seconds.

  • acquire_timeout_secs: Timeout for acquiring a connection from the pool (default: 5)
  • idle_timeout_secs: Timeout before closing idle connections (default: 600)
  • max_lifetime_secs: Maximum lifetime of a connection (default: 1800)

Rust Example with Custom Timeouts

let register = Register::new(
    connection_string,
    table_name,
    id_column,
    jsonb_column,
    pool_size,
    lru_cache_size,
    Some(10),   // 10 second acquire timeout
    Some(300),  // 5 minute idle timeout
    Some(3600), // 1 hour max lifetime
).await?;

Python Example with Custom Timeouts

register = JsonRegister(
    database_name="dbname",
    database_host="localhost",
    database_port=5432,
    database_user="user",
    database_password="password",
    acquire_timeout_secs=10,   # 10 second acquire timeout
    idle_timeout_secs=300,     # 5 minute idle timeout
    max_lifetime_secs=3600,    # 1 hour max lifetime
)

Monitoring

The library provides comprehensive telemetry metrics for integration with monitoring systems such as Prometheus, OpenTelemetry, or custom logging. All metrics can be retrieved individually or as a complete snapshot.

Connection Pool Metrics

  • pool_size(): Total number of connections in the pool (idle and active)
  • idle_connections(): Number of idle connections available for use
  • active_connections(): Number of connections currently in use
  • is_closed(): Whether the connection pool is closed

Cache Metrics

  • cache_hits(): Total number of successful cache lookups
  • cache_misses(): Total number of unsuccessful cache lookups
  • cache_hit_rate(): Hit rate as a percentage (0.0 to 100.0)
  • cache_size(): Current number of items in the cache
  • cache_capacity(): Maximum cache capacity
  • cache_evictions(): Total number of items evicted from the cache

Database Metrics

  • db_queries_total(): Total number of database queries executed
  • db_query_errors(): Total number of failed database queries

Operation Metrics

  • register_single_calls(): Number of times register_object was called
  • register_batch_calls(): Number of times register_batch_objects was called
  • total_objects_registered(): Total number of objects registered across all calls

Telemetry Snapshot

The telemetry_metrics() method (Rust only) returns a complete snapshot of all metrics in a single call, which is useful for OpenTelemetry exporters

Rust Monitoring Example

// Get all metrics at once (recommended for OpenTelemetry)
let metrics = register.telemetry_metrics();
println!("Cache: {}/{} items, {} evictions", metrics.cache_size, metrics.cache_capacity, metrics.cache_evictions);
println!("Cache performance: {} hits, {} misses ({:.2}% hit rate)",
    metrics.cache_hits, metrics.cache_misses, metrics.cache_hit_rate);
println!("Pool: {} total, {} active, {} idle",
    metrics.pool_size, metrics.active_connections, metrics.idle_connections);
println!("Database: {} queries, {} errors",
    metrics.db_queries_total, metrics.db_query_errors);
println!("Operations: {} objects registered ({} single + {} batch calls)",
    metrics.total_objects_registered, metrics.register_single_calls, metrics.register_batch_calls);

// Or query individual metrics
let hit_rate = register.cache_hit_rate();
let active = register.active_connections();

Python Monitoring Example

# Individual metrics
print(f"Cache: {register.cache_size()}/{register.cache_capacity()} items")
print(f"Cache evictions: {register.cache_evictions()}")
print(f"Active connections: {register.active_connections()}")
print(f"DB queries: {register.db_queries_total()}, errors: {register.db_query_errors()}")
print(f"Objects registered: {register.total_objects_registered()}")
print(f"Single calls: {register.register_single_calls()}, Batch calls: {register.register_batch_calls()}")

License

This project is licensed under the Apache-2.0 License.