lutra-runner-duckdb

DuckDB runner implementation for Lutra - enables in-process analytical queries.

Overview

lutra-runner-duckdb implements the lutra_runner::RunSync trait, providing an embedded analytical database using DuckDB. This allows you to execute Lutra programs in-process with DuckDB's fast analytical query engine.

Key Features:

In-memory or file-based database
Synchronous execution (no async runtime required)
Full support for primitives, tuples, arrays, and enums (including nested types).
No external database server required

Installation

Add to your Cargo.toml:

[dependencies]
lutra-runner-duckdb = "0.3"

DuckDB Version and Bundling

This crate uses the duckdb crate, which requires DuckDB library to be provided in one of the following ways:

Dev: Uses nixpkgs.duckdb from the nix environment and link dynamically.
Prod - dynamic linking: Requires DuckDB shared library (.so/.dylib/.dll) available in the build environment.
Prod - static linking: The duckdb crate supports a bundled feature that bundles DuckDB into this library, avoiding the need for a system library. To use this, add the feature to your dependencies:
```
[dependencies]
duckdb = { version = "1.4", features = ["bundled"] }
```

Usage

Rust API

Synchronous Usage

use lutra_runner::RunSync;
use lutra_runner_duckdb::Runner;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create in-memory database
    let mut runner = Runner::in_memory(None)?;
    
    // Or use file-based storage
    let mut runner = Runner::open("data/analysis.duckdb", None)?;
    
    // Prepare and execute a program
    let prepared = runner.prepare_sync(program)?;
    let output = runner.execute_sync(&prepared, &input)?;
    
    Ok(())
}

Async Usage

If you need to use the runner in an async context, wrap it with AsyncRunner:

use lutra_runner::{Run, AsyncRunner};
use lutra_runner_duckdb::Runner;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create runner and wrap for async usage
    let sync_runner = Runner::in_memory(None)?;
    let runner = AsyncRunner::new(sync_runner);
    
    // Prepare and execute a program
    let prepared = runner.prepare(program).await?;
    let output = runner.execute(&prepared, &input).await?;
    
    Ok(())
}

CLI

Lutra CLI provides this runner via --duckdb argument.

# In-memory database
lutra run --duckdb=:memory:

# File-based database
lutra run --duckdb=data.duckdb

Work in progress

Following features are currently being worked on.

1. Arrays/Enums from External Tables

When constructing arrays in Lutra code, they work perfectly (see examples below). However, reading arrays/enums from existing DuckDB table columns via std::sql::from() is not yet fully supported:

// ✅ Works - constructing arrays in Lutra
func main() -> [1, 2, 3]: [int64]

// ✅ Works - arrays in tuples (serialized as DuckDB LIST via subquery)
func main() -> {nums: [1, 2, 3]: [int64], count: 3}

// ⚠️ Limited - reading LIST columns from existing tables
func process_data() -> std::sql::from("table_with_list_column")
// This may not deserialize LIST columns correctly

2. Schema Introspection

The get_interface() method currently returns an empty string. Schema introspection from existing DuckDB databases is not yet implemented.

3. Input Parameter Limitations

Arrays of tuples as input parameters don't work because duckdb-rs does not yet support STRUCT and LIST values in their bind_parameter function.

Contributing

See the main Lutra repository for contribution guidelines.

License

This project is part of the Lutra language project. See the main repository for license information.

lutra-runner-duckdb 0.5.0