Crate vgi 

Build native, single-binary DuckDB extensions in Rust — no C++, no linking against DuckDB.

vgi is the Rust SDK for writing VGI (Vector Gateway Interface) workers: the worker side of Query Farm’s DuckDB “Hyperfederation” extension. A worker is an ordinary Rust binary that DuckDB launches and talks to over Apache Arrow IPC. It exposes scalar / table / aggregate functions and whole catalogs (schemas, tables, views) that behave like native DuckDB objects — with no compiled C++ extension and no version coupling to a specific DuckDB build.

Workers built with this crate are byte-for-byte wire-compatible with the canonical Python implementation, so a Rust worker drops in behind the same ATTACH … (TYPE vgi). It is built on the vgi-rpc crate (wire protocol, RPC server, transports), uses stock arrow-rs 59.x, and has an MSRV of 1.97.

Your first worker

A worker is a main() that registers functions on a Worker and calls Worker::run. This one exposes upper_case(varchar) -> varchar:

use std::sync::Arc;

use arrow_array::{cast::AsArray, ArrayRef, RecordBatch, StringArray};
use arrow_schema::DataType;
use vgi::{ArgSpec, FunctionMetadata, ProcessParams, ScalarFunction, Worker};
use vgi_rpc::{Result, RpcError};

/// `upper_case(s)` — uppercase a string column.
struct UpperCase;

impl ScalarFunction for UpperCase {
    fn name(&self) -> &str {
        "upper_case"
    }

    fn metadata(&self) -> FunctionMetadata {
        FunctionMetadata {
            description: "Convert string values to uppercase".into(),
            return_type: Some(DataType::Utf8),
            ..Default::default()
        }
    }

    fn argument_specs(&self) -> Vec<ArgSpec> {
        vec![ArgSpec::column("value", 0, "varchar", "String to uppercase")]
    }

    fn process(&self, params: &ProcessParams, batch: &RecordBatch) -> Result<RecordBatch> {
        let col = batch.column(0).as_string::<i32>();
        let upper: StringArray = col.iter().map(|v| v.map(str::to_uppercase)).collect();
        let out: ArrayRef = Arc::new(upper);
        RecordBatch::try_new(params.output_schema.clone(), vec![out])
            .map_err(|e| RpcError::runtime_error(e.to_string()))
    }
}

fn main() {
    let mut worker = Worker::new();
    worker.register_scalar(UpperCase);
    worker.run(); // serves stdio (default), --unix <path>, or --http
}

Build it (cargo build --release), then call it from a DuckDB engine that has the vgi extension — Query Farm’s Haybarn distribution ships it and starts with uvx haybarn-cli:

ATTACH 'demo' (TYPE vgi, LOCATION './target/release/my-worker');
SELECT demo.main.upper_case(name) FROM (VALUES ('alice'), ('bob')) t(name);
-- ALICE
-- BOB

The function model

Implement one trait per function kind and register it on the Worker:

Kind	Trait	Use case
Scalar	ScalarFunction	Per-row transforms (1 row in → 1 row out)
Table	table_function::TableFunction	Generate / scan rows (no row input)
Table-in-out	table_in_out::TableInOutFunction	Streaming row transforms (N in → M out)
Buffering	buffering::TableBufferingFunction	Sink → combine → source (aggregate-emit)
Aggregate	aggregate::AggregateFunction	Grouped / window / streaming aggregates

Every trait shares the same bind/process vocabulary: ArgSpec declares the arguments, FunctionMetadata declares optimizer-facing properties, BindParams / BindResponse resolve the output schema at bind time, and ProcessParams carries per-call context (settings, secrets, pushdown hints) into the work method.

Beyond functions

Worker::set_catalog exposes a full catalog — schemas, function-backed tables, views, and macros — with constraints, column statistics, time travel (AT), and secondary catalogs attachable by name (see catalog). Projection and filter pushdown, ORDER BY / TABLESAMPLE hints, custom settings, secrets, and bearer auth are handled for you.

Transports

Worker::run selects a transport from argv: stdio (default), Unix socket (--unix <path>, the launcher contract), or HTTP (--http, Arrow-IPC over HTTP with AEAD-sealed stateless stream tokens and optional bearer auth). You rarely pass these yourself — DuckDB supplies the right flags when it launches your worker.

Re-exports

pub use dispatch::FunctionScope;

pub use function::ArgSpec;

pub use function::ArgumentMonotonicity;

pub use function::BindParams;

pub use function::BindResponse;

pub use function::FilterHandling;

pub use function::FunctionMetadata;

pub use function::ProcessParams;

pub use function::ScalarFunction;

pub use worker::HostedProtocolsHook;

pub use worker::ServeTransport;

pub use worker::Worker;

pub use worker::INTROSPECT_PRINCIPALS_ENV;

pub use ::vgi_rpc;

Modules

aggregate

Aggregate function model (update / combine / finalize).

arguments

Parsing of the arguments wire blob.

attach_ticket

Attach tickets: a user’s ATTACH, sealed so a runner can replay it later as that user (vgi.attach_tickets.v1).

buffering

Table buffering (sink + source) function model.

cache_control

Result-cache control metadata (vgi.cache.*).

catalog

Default read-only catalog: auto-generates SchemaInfo + FunctionInfo from the worker’s registered functions.

catalog_contents

catalog_contents (protocol 2.1.0): the whole catalog in one call, with etag revalidation.

catalog_snapshot

Opt-in, validated publications of committed catalog metadata. Attachment authorization is performed by the dispatcher before this layer.

catalog_snapshot_s3

S3/R2 publication repository with conditional manifests and immutable objects.

column_expressions

Canonical VIRTUAL generated-column carriers.

column_metadata

constraints

Canonical constraint records for catalogs that state their own facts.

copy_from

Custom COPY ... FROM format readers.

copy_to

Custom COPY ... TO format writers.

diagnostics

Execution-scoped diagnostics delivered alongside a worker’s Arrow results.

dispatch

The VGI dispatcher: owns the function registries + catalog identity and implements every RPC handler (bind, init, and the catalog discovery methods).

errors

Coded errors: tell “your input was wrong” apart from “the worker broke”.

function

Core function model shared by all VGI function kinds.

ipc

Arrow IPC stream helpers for the binary-valued wire fields.

numeric

Numeric scalar helpers (port of Python _promote_for_addition + NumericDispatch).

opaque

Sealing attach_opaque_data and transaction_opaque_data.

overload

Function overload resolution.

partition

Partition-column support: mark schema fields as partition columns and compute the per-batch vgi_partition_values#b64 metadata (base64 of a 2-row min/max IPC batch) that the C++ extension reads to plan partitioned aggregates. Mirrors the canonical Python partition_field.

projection

protocol

VGI wire protocol: DTOs, enum payloads, and RPC method registration.

pushdown

Strict consumer and Arrow evaluator for VGI Filter Encoding v2.

secrets

Secret access for functions.

settings

DuckDB session settings passed to functions.

split_token

Split-token envelope: the framework’s wrapper around a worker’s split payload.

statistics

Column-statistics serialization: the sparse-union IPC batch DuckDB’s VGI extension reads to seed the optimizer. Mirrors the canonical Python serialize_column_statistics.

storage

Cross-process state storage for VGI workers.

stored_catalog

DDL-capable catalogs whose state lives in the worker’s shared storage.

table_function

Table (producer) function model: generate output batches without input.

table_in_out

Table-in-out function model: transform input batches to output batches.

transport

Worker transport selection: stdio (default), AF_UNIX (launcher), TCP, HTTP.

wasm_worker

Canonical browser (worker:) entry points for a VGI worker.

wire

Flat wire serialization for VGI protocol DTOs.

worker

The VGI worker: owns function registries (via Dispatcher), builds the RPC server, and drives transport selection from argv.

Macros

resume_fields

Implement the three TableProducer resume hooks from a list of fields.

wasm_worker

Generate the canonical browser worker: entry points for a VGI worker.

Structs

FunctionExample

One examples entry of FunctionInfo.

RpcError

An RPC-level error, serialized on the wire as an EXCEPTION log batch.

Enums

BindRepeatability

Equivalent scans of one exact bound state within a query. This does not promise identical independent binds or later queries.

Constants

VGI_PROTOCOL_NAME

RPC protocol name — the vgi_rpc.protocol routing key every VGI request carries, and the name this worker advertises as hosting.

VGI_PROTOCOL_VERSION

VGI wire protocol version advertised to the C++ extension.

Type Aliases

Result

Convenience alias for Result<T, RpcError>.