Crate vllm_cpp 

Safe model inference API for the stable vllm.cpp C boundary.

Entry points

Resolve a Hub model with HuggingFaceModel (default main, or an explicit revision), then create an Engine with Engine::load or configure native model settings through EngineBuilder. SamplingParams owns sampling, stop-string, StructuredOutput settings, and an optional host-side logits processor for completion calls. The engine provides blocking completion, streaming, raw-JSON chat, and Engine::submit for a concurrent Request. Enable serde for serde_json::Value chat helpers.

Ownership and callbacks

Engine is a cloneable RAII owner; clones share one reference-counted native engine. Rust copies completion, stream, chat, and error text before native storage is freed or reused. Blocking callbacks may borrow caller data. Their panics are caught before the C boundary and resumed after the native call returns. Custom logits processors are Send + Sync, may run concurrently on native worker threads, and report contained panic through Error::LogitsProcessorPanicked.

A Request retains its engine and asynchronous callback until native free/join completes. Requests are Send but intentionally not Sync, while engines are Send + Sync. Asynchronous callbacks run on a native delivery thread, must be Send + 'static, and surface panic through Error::CallbackPanicked. ABI version 10 forbids waiting for or freeing a request from its callback thread; callback-thread drop delegates ownership to a cleanup reaper instead.

ABI, linking, and deployment

Engine loading requires the linked native library’s ABI to equal expected_abi_version before versioned structs cross FFI. version copies the linked library’s diagnostic version string. The default bundled feature builds the pinned native source. system selects a caller-provided installation, dynamic-link selects shared linking, and serde adds typed JSON helpers. The non-optional hf-hub dependency provides synchronous, cache-aware model retrieval without an async runtime. CUDA, CUTLASS, Triton AOT, Vulkan, Metal, and external MLX features are experimental bundled build configuration.

Dynamic linking does not deploy libvllm.so or libvllm.dylib; applications must make it and its runtime dependencies visible through the platform loader, such as LD_LIBRARY_PATH, DYLD_LIBRARY_PATH, or an application-owned rpath. The supported runtime tier is native Linux x86_64 CPU. Accelerator features are build/configuration surfaces with known runtime blockers, not complete accelerator runtime support.

Structs

Completion

A Rust-owned blocking completion result.

Engine

A cloneable vllm.cpp serving engine.

EngineBuilder

Builder for one complete serving engine.

HuggingFaceModel

A synchronous Hugging Face model resolver.

Request

An owned non-blocking streaming request.

SamplingParams

Owned sampling configuration for one generation request.

StreamEvent

One copied streaming delta.

StreamOutcome

How a successful blocking stream ended.

Enums

Error

An error returned by the safe vllm.cpp wrapper.

FinishReason

Why native generation finished.

HuggingFaceError

An error returned while resolving a model from the Hugging Face Hub.

RequestOutcome

How a successfully waited non-blocking request ended.

SchedulerPolicy

Native scheduler admission order.

StreamControl

Controls whether native streaming continues after a callback.

StructuredOutput

One engine-side structured decoding constraint.

Toggle

A native tri-state setting whose default is resolved by vllm.cpp.

Functions

abi_version

Returns the C ABI reported by the linked vllm.cpp library.

expected_abi_version

Returns the compile-time C ABI expected by this crate.

version

Copies the version string reported by the linked vllm.cpp library.