Crate algeon_datafusion 

Algeon for DataFusion

algeon-datafusion runs supported Apache DataFusion plans on NVIDIA GPUs through RAPIDS. A physical optimizer rule replaces whole relational plan candidates with cuDF execution and leaves the rest of the plan on the DataFusion CPU path. SQL table functions expose cuGraph, cuVS and cuML algorithms, so one query can select an algorithm’s input, run it and join the result. Setup, supported SQL, configuration and design are documented at algeon.dev. Algeon is experimental; APIs and query behavior may change between releases.

Dependencies

[dependencies]
algeon-datafusion = "0.1.1"
datafusion = "55.1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
futures = "0.3"

Feature	Adds	Native libraries
none	Relational execution on cuDF	cuDF, RMM
cuvs	Vector search and clustering table functions	cuVS
cugraph	Graph table functions	cuVS, cuGraph
cuml	Machine learning and forest inference table functions	cuVS, cuML
server	Embedded Arrow Flight SQL server	none
nvml	Live device diagnostics through NVML	none

No feature is on by default. Algeon needs Linux, an NVIDIA GPU and a CUDA toolkit, and Cargo does not build the RAPIDS libraries. Build them from the Algeon source tag that matches the crate version in the application’s Cargo.lock (v0.1.1 for 0.1.1) and point ALGEON_NATIVE_DIR and LD_LIBRARY_PATH at the installed prefixes. Building for an application gives the build order per feature and the loader setup; Prerequisites covers the driver, toolkit and host compiler.

Example

use algeon_datafusion::{
    AlgeonGpuBackend, AlgeonGpuDeviceConfig, AlgeonGpuDeviceProfile, AttemptOptions, GpuMemoryOwnership,
};
use datafusion::{execution::SessionStateBuilder, prelude::SessionContext};
use futures::TryStreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Device 0 with process-wide allocator ownership; no memory is allocated here.
    let device = AlgeonGpuDeviceConfig::new(
        AlgeonGpuDeviceProfile::new(0),
        GpuMemoryOwnership::WholeDeviceExclusive,
    );
    let backend = AlgeonGpuBackend::builder().devices([device]).build()?;
    let state = backend.install_on(SessionStateBuilder::new_with_default_features())?.build();
    let ctx = SessionContext::new_with_state(state);
    let attempt = backend.begin_attempt(AttemptOptions::default())?;
    // Taken before planning so the terminal report is observable even if planning fails.
    let completion = attempt.completion();
    let batches = attempt.sql(&ctx, "SELECT 1 AS value").await?.try_collect::<Vec<_>>().await?;
    let report = completion.await;
    println!("{} batch(es), outcome {:?}", batches.len(), report.outcome);
    backend.close_and_wait_async(std::time::Duration::from_secs(10)).await?;
    Ok(())
}

DataFusion session walks through the same steps, adds the algorithm SQL families and explains execution modes.

Entry points

Step	Type	Module
Build one backend per process	AlgeonGpuBackend	backend
Install it on a session builder	AlgeonGpuBackend::install_on, AlgeonSessionStateBuilderExt	session
Run one query	AlgeonAttempt	attempt

The backend builder fixes the device envelope at construction. Each attempt plans and executes one query, then publishes a terminal report through AlgeonAttempt::completion. Under AlgeonGpuExecutionMode::NativeRequired the optimizer rejects a completed plan that still executes DataFusion CPU operators.

Other modules

Purpose	Modules
SQL table functions	cugraph_sql, cuvs_sql, cuml_sql, gpu_functions
Reports and diagnostics	report, native_report, gpu_coverage, observability, build_info
Errors and reason codes	error, capability
Plan structures exposed for tooling	planner, native_exec, source_cardinality
Flight SQL server, feature server	server

Features

The table under Dependencies lists what each feature adds and the native libraries it links. With a feature disabled:

• cugraph: cuGraph table functions fail closed with a typed capability error.
• cuvs, cuml: the SQL family stays discoverable and execution fails before input staging.
• nvml: device telemetry reports FeatureDisabled for NVML and leaves the NVML values None. CUDA telemetry, admission and memory accounting do not change.
• server: the server module is absent.

bench-support and qualification serve tooling in the Algeon repository. The first adds application and test support outside the embedded and server report contract; the second closes admission on one live device ledger for the multi-GPU validation tool.

Re-exports

pub use attempt::AlgeonAttempt;

pub use attempt::AttemptOptions;

pub use backend::AlgeonGpuBackend;

Modules

attempt

Explicit execution-attempt adapter boundary.

backend

Process-scoped GPU backend facade for embedded DataFusion sessions.

build_info

Build identity metadata including version, commit, optional engines, and the linked DataFusion release.

capability

CPU/GPU candidate-outcome capability reason codes.

cugraph_sql

Relation-aware cuGraph SQL planning and execution contracts.

cuml_sql

Relation-aware cuML SQL planning, catalog metadata, and execution.

cuvs_sql

Relation-aware cuVS SQL planning and execution contracts.

error

One error taxonomy for the DataFusion adapter boundary.

gpu_coverage

Query-level GPU coverage validation.

gpu_functions

Common DataFusion-facing GPU UDTF discovery and validation catalog.

native_exec

DataFusion ExecutionPlan wrappers around native cuDF query plans.

native_report

Public report and TSV contracts for native planning/execution diagnostics.

observability

Stable backend observability contracts for application exporters.

planner

DataFusion physical optimizer integration for native cuDF execution.

report

Planning report structures for capability, cost-selection, and final-plan evidence.

serverserver

Opt-in Flight SQL server for algeon-datafusion.

session

Installation paths for caller-owned DataFusion session builders.

source_cardinality

Source cardinality proofs from Arrow schema metadata.

Structs

AlgeonGpuDeviceConfig

Canonical device profile bound to an explicit process-scoped GPU memory ownership promise.

AlgeonGpuDeviceProfile

Canonical device profile bound to an explicit process-scoped GPU memory ownership promise.

Enums

GpuMemoryOwnership

Deployment ownership promised for one CUDA device.