vllm_cpp/
lib.rs

1//! Safe model inference API for the stable vllm.cpp C boundary.
2//!
3//! # Entry points
4//!
5//! Resolve a Hub model with [`HuggingFaceModel`] (default `main`, or an explicit
6//! revision), then create an [`Engine`] with [`Engine::load`] or configure native
7//! model settings through [`EngineBuilder`]. [`SamplingParams`] owns sampling,
8//! stop-string, [`StructuredOutput`] settings, and an optional host-side logits
9//! processor for completion calls. The engine provides
10//! blocking completion, streaming, raw-JSON chat, and [`Engine::submit`] for a
11//! concurrent [`Request`]. Enable `serde` for `serde_json::Value` chat helpers.
12//!
13//! # Ownership and callbacks
14//!
15//! [`Engine`] is a cloneable RAII owner; clones share one reference-counted
16//! native engine. Rust copies completion, stream, chat, and error text before
17//! native storage is freed or reused. Blocking callbacks may borrow caller data.
18//! Their panics are caught before the C boundary and resumed after the native
19//! call returns. Custom logits processors are `Send + Sync`, may run concurrently
20//! on native worker threads, and report contained panic through
21//! [`Error::LogitsProcessorPanicked`].
22//!
23//! A [`Request`] retains its engine and asynchronous callback until native
24//! free/join completes. Requests are `Send` but intentionally not `Sync`, while
25//! engines are `Send + Sync`. Asynchronous callbacks run on a native delivery
26//! thread, must be `Send + 'static`, and surface panic through
27//! [`Error::CallbackPanicked`]. ABI version 10 forbids waiting for or freeing a
28//! request from its callback thread; callback-thread drop delegates ownership to
29//! a cleanup reaper instead.
30//!
31//! # ABI, linking, and deployment
32//!
33//! Engine loading requires the linked native library's ABI to equal
34//! [`expected_abi_version`] before versioned structs cross FFI. [`version`] copies
35//! the linked library's diagnostic version string. The default
36//! `bundled` feature builds the pinned native source. `system` selects a
37//! caller-provided installation, `dynamic-link` selects shared linking, and
38//! `serde` adds typed JSON helpers. The non-optional `hf-hub` dependency provides
39//! synchronous, cache-aware model retrieval without an async runtime. CUDA,
40//! CUTLASS, Triton AOT, Vulkan, Metal, and external MLX features are experimental
41//! bundled build configuration.
42//!
43//! Dynamic linking does not deploy `libvllm.so` or `libvllm.dylib`; applications
44//! must make it and its runtime dependencies visible through the platform loader,
45//! such as `LD_LIBRARY_PATH`, `DYLD_LIBRARY_PATH`, or an application-owned rpath.
46//! The supported runtime tier is native Linux x86_64 CPU.
47//! Accelerator features are build/configuration surfaces with known runtime
48//! blockers, not complete accelerator runtime support.
49
50mod callback;
51mod engine;
52mod error;
53mod hf;
54mod params;
55mod request;
56
57pub use callback::{StreamControl, StreamEvent, StreamOutcome};
58pub use engine::{Completion, Engine, EngineBuilder, FinishReason};
59pub use error::{Error, HuggingFaceError};
60pub use hf::HuggingFaceModel;
61pub use params::{SamplingParams, SchedulerPolicy, StructuredOutput, Toggle};
62pub use request::{Request, RequestOutcome};
63
64/// Returns the compile-time C ABI expected by this crate.
65#[must_use]
66pub const fn expected_abi_version() -> i32 {
67    vllm_cpp_sys::VLLM_ABI_VERSION as i32
68}
69
70/// Returns the C ABI reported by the linked vllm.cpp library.
71///
72/// Engine loading compares this value for exact equality before passing any
73/// versioned native struct.
74#[must_use]
75pub fn abi_version() -> i32 {
76    // SAFETY: this base ABI function takes no pointers and returns a plain i32.
77    unsafe { vllm_cpp_sys::vllm_abi_version() }
78}
79
80/// Copies the version string reported by the linked vllm.cpp library.
81///
82/// This diagnostic does not replace [`abi_version`]: callers must still use the
83/// numeric ABI for compatibility decisions.
84pub fn version() -> Result<String, Error> {
85    // SAFETY: the base ABI returns a borrowed, process-lifetime C string.
86    let pointer = unsafe { vllm_cpp_sys::vllm_version() };
87    if pointer.is_null() {
88        return Err(Error::Runtime {
89            message: "vllm_version returned a null pointer".to_owned(),
90        });
91    }
92    // SAFETY: the native contract returns a live NUL-terminated string.
93    unsafe { std::ffi::CStr::from_ptr(pointer) }
94        .to_str()
95        .map(str::to_owned)
96        .map_err(|_| Error::InvalidUtf8 {
97            field: "native version",
98        })
99}
vllm_cpp/lib.rs

vllm_cpp/
lib.rs
