Crate polars_redis 

polars-redis

Query Redis like a database. Transform with Polars. Write back without ETL.

This crate provides a Redis IO plugin for Polars, enabling you to scan Redis data structures as Arrow RecordBatches with support for projection pushdown and batched iteration.

Supported Redis Types

Type	Read	Write	Description
Hash	Yes	Yes	Field-level projection pushdown
JSON	Yes	Yes	RedisJSON documents
String	Yes	Yes	Simple key-value pairs
Set	Yes	Yes	Unique members
List	Yes	Yes	Ordered elements
Sorted Set	Yes	Yes	Members with scores
Stream	Yes	No	Timestamped entries
TimeSeries	Yes	No	Server-side aggregation

Quick Start

Reading Hashes

use polars_redis::{HashBatchIterator, HashSchema, BatchConfig, RedisType};

// Define schema for hash fields
let schema = HashSchema::new(vec![
    ("name".to_string(), RedisType::Utf8),
    ("age".to_string(), RedisType::Int64),
    ("active".to_string(), RedisType::Boolean),
])
.with_key(true)
.with_key_column_name("_key".to_string());

// Configure batch iteration
let config = BatchConfig::new("user:*".to_string())
    .with_batch_size(1000)
    .with_count_hint(100);

// Create iterator
let mut iterator = HashBatchIterator::new(
    "redis://localhost:6379",
    schema,
    config,
    None, // projection
).unwrap();

// Iterate over batches
while let Some(batch) = iterator.next_batch().unwrap() {
    println!("Got {} rows", batch.num_rows());
}

Writing Hashes

use polars_redis::{write_hashes, WriteMode};

let keys = vec!["user:1".to_string(), "user:2".to_string()];
let fields = vec!["name".to_string(), "age".to_string()];
let values = vec![
    vec![Some("Alice".to_string()), Some("30".to_string())],
    vec![Some("Bob".to_string()), Some("25".to_string())],
];

let result = write_hashes(
    "redis://localhost:6379",
    keys,
    fields,
    values,
    Some(3600), // TTL in seconds
    WriteMode::Replace,
).unwrap();

println!("Wrote {} keys", result.keys_written);

Schema Inference

use polars_redis::infer_hash_schema;

// Sample keys to infer schema
let schema = infer_hash_schema(
    "redis://localhost:6379",
    "user:*",
    100,  // sample size
    true, // type inference
).unwrap();

for (name, dtype) in schema.fields {
    println!("{}: {:?}", name, dtype);
}

Python Bindings

This crate also provides Python bindings via PyO3 when built with the python feature. The Python package polars-redis wraps these bindings with a high-level API.

Features

• python - Enable Python bindings (PyO3)
• json - Enable RedisJSON support (enabled by default)
• search - Enable RediSearch support (enabled by default)
• cluster - Enable Redis Cluster support

Re-exports

pub use options::HashScanOptions;

pub use options::JsonScanOptions;

pub use options::KeyColumn;

pub use options::ParallelStrategy;

pub use options::RowIndex;

pub use options::RowIndexColumn;

pub use options::ScanOptions;

pub use options::StreamScanOptions;

pub use options::StringScanOptions;

pub use options::TimeSeriesScanOptions;

pub use options::TtlColumn;

pub use options::get_default_batch_size;

pub use options::get_default_count_hint;

pub use options::get_default_timeout_ms;

pub use parallel::FetchResult;

pub use parallel::KeyBatch;

pub use parallel::ParallelConfig;

pub use parallel::ParallelFetch;

pub use query_builder::Predicate;

pub use query_builder::PredicateBuilder;

pub use query_builder::Value;

Modules

cache

DataFrame caching in Redis using Arrow IPC or Parquet format.

options

Configuration options for Redis operations.

parallel

Parallel batch fetching infrastructure.

pubsub

Redis Pub/Sub support for collecting messages into Arrow RecordBatches.

query_builder

Query builder for translating filter expressions to RediSearch queries.

search

RediSearch integration for server-side filtering.

Structs

BatchConfig

Configuration for batch iteration.

FieldInferenceInfo

Detailed inference information for a single field.

HashBatchIterator

Iterator for scanning Redis hashes in batches as Arrow RecordBatches.

HashFetcher

Fetcher for parallel hash operations.

HashSchema

Schema for a Redis hash, mapping field names to types.

HashSearchIterator

Iterator for RediSearch FT.SEARCH results.

InferredSchema

Inferred schema from Redis data.

InferredSchemaWithConfidence

Inferred schema with detailed confidence information.

JsonBatchIterator

Iterator for scanning RedisJSON documents in batches as Arrow RecordBatches.

JsonSchema

Schema for RedisJSON documents, mapping JSONPath expressions to Arrow types.

KeyError

Error information for a single key.

ListBatchIterator

Iterator for scanning Redis lists in batches as Arrow RecordBatches.

ListSchema

Schema configuration for Redis list scanning.

RedisConnection

Redis connection wrapper that manages connection lifecycle.

SearchBatchConfig

Configuration for search-based batch iteration.

SetBatchIterator

Iterator for scanning Redis sets in batches as Arrow RecordBatches.

SetSchema

Schema configuration for Redis set scanning.

StreamBatchIterator

Iterator for scanning Redis Streams in batches as Arrow RecordBatches.

StreamSchema

Schema configuration for Redis Stream scanning.

StringBatchIterator

Iterator for scanning Redis strings in batches as Arrow RecordBatches.

StringSchema

Schema for Redis string values, defining how to parse and convert them.

TimeSeriesBatchIterator

Iterator for scanning RedisTimeSeries in batches as Arrow RecordBatches.

TimeSeriesSchema

Schema configuration for RedisTimeSeries scanning.

WriteResult

Result of a write operation (basic).

WriteResultDetailed

Detailed result of a write operation with per-key error information.

ZSetBatchIterator

Iterator for scanning Redis sorted sets in batches as Arrow RecordBatches.

ZSetSchema

Schema configuration for Redis sorted set scanning.

Enums

ConnectionConfig

Configuration for Redis connections.

Error

Errors that can occur in polars-redis operations.

RedisConn

Unified async connection that works for both single-node and cluster.

RedisType

Supported data types for Redis field conversion.

WriteMode

Write mode for handling existing keys.

Functions

batch_to_ipc

Serialize an Arrow RecordBatch to IPC format bytes.

infer_hash_schema

Infer schema from Redis hashes.

infer_hash_schema_with_confidence

Infer schema from Redis hashes with detailed confidence information.

infer_json_schema

Infer schema from RedisJSON documents.

write_hashes

Write hashes to Redis from field data.

write_hashes_detailed

Write hashes to Redis with detailed per-key error reporting.

write_json

Write JSON documents to Redis.

write_lists

Write list elements to Redis.

write_sets

Write set members to Redis.

write_strings

Write string values to Redis.

write_zsets

Write sorted set members to Redis.

Type Aliases

Result

Result type alias for polars-redis operations.