# zvec-rust
[](https://github.com/zvec-ai/zvec-rust/actions/workflows/ci.yml)
[](https://crates.io/crates/zvec-rust)
[](LICENSE)
Safe, idiomatic Rust bindings for the [zvec](https://github.com/alibaba/zvec) vector database.
## Features
- **RAII Resource Management** — All C resources are automatically freed via `Drop`
- **Builder Pattern** — Fluent APIs for schema, query, and configuration
- **Type Safety** — Rust enums for all C constants with compile-time checks
- **Comprehensive Error Handling** — All FFI calls return `Result<T>` with detailed error codes
- **Zero-Copy Where Possible** — Minimizes data copying across the FFI boundary
- **Prebuilt Libraries** — Automatically downloads prebuilt `libzvec_c_api` from GitHub Releases; advanced users can override with `ZVEC_LIB_DIR`
- **FTS with jieba Out of the Box** — `libzvec_c_api` embeds cppjieba's default dictionary and HMM model (LZ4-compressed), so the `jieba` FTS tokenizer needs no dictionary files on disk and no registration step
## Supported Platforms
| **macOS** | ARM64 (Apple Silicon) | ✅ Prebuilt + Clippy + Test | Primary development platform; requires macOS 11+ |
| **macOS** | x86_64 (Intel) | ✅ Prebuilt + Clippy + Test | Built natively on `macos-15-intel`; requires macOS 11+ |
| **Linux** | x86_64 | ✅ Clippy + Test + Fuzz + Coverage + Benchmark | Full CI coverage |
| **Linux** | ARM64 (AArch64) | ✅ Clippy + Test + Fuzz + Coverage | |
| **Linux (musl)** | x86_64 | ✅ Prebuilt | Alpine/musl distros; built in `musllinux_1_2`, same baseline as upstream zvec |
| **Linux (musl)** | ARM64 (AArch64) | ✅ Prebuilt | Alpine/musl distros; built in `musllinux_1_2` |
| **Windows** | x86_64 (MSVC) | ✅ Prebuilt + Clippy + Test | CMake + MSVC toolchain |
| **Windows** | ARM64 (MSVC) | ✅ Prebuilt | Built natively on `windows-11-arm`, the runner and MSVC arch upstream zvec uses |
> Linux gnu prebuilts target glibc 2.28 (`manylinux_2_28`, same as upstream
> zvec), so they work on glibc distros as old as CentOS 8 / Ubuntu 20.04.
> musl prebuilts work on musl distros such as Alpine (musl 1.2+).
> macOS prebuilts target macOS 11.0 on both architectures, the same baseline as
> upstream zvec's SDK builds. Releases up to v0.7.2 stamped macOS 14.0 on Apple
> Silicon and shipped no Intel build.
> The dynamic library name varies by platform: `libzvec_c_api.dylib` (macOS), `libzvec_c_api.so` (Linux), `zvec_c_api.dll` (Windows).
> The Windows ARM64 prebuilt ships without RaBitQ and DiskANN: upstream zvec
> gates RaBitQ on x86_64 (its kernels are AVX2/AVX-512) and its Windows
> DiskANN backend on x86_64 too, so those index types are unavailable on that
> target.
## Architecture
```
zvec-rust/
├── zvec-sys/ # Low-level FFI bindings to libzvec_c_api
├── zvec/ # Safe, high-level Rust wrapper
└── fuzz/ # Fuzz testing targets
```
- **`zvec-rust-sys`** — Raw `extern "C"` declarations, opaque pointer types, and constants
- **`zvec-rust`** — Safe wrappers with RAII, builders, iterators, and idiomatic Rust APIs
## Prerequisites
The Rust SDK depends on the zvec C library (`libzvec_c_api`). Choose one of the following ways to provide it:
### Option 1: Bundled Prebuilt Library (Zero Setup)
Add `zvec-rust` to your `Cargo.toml`. The default `bundled` feature automatically downloads the prebuilt `libzvec_c_api` for your platform from [GitHub Releases](https://github.com/zvec-ai/zvec-rust/releases):
```toml
[dependencies]
zvec-rust = "0.8.0"
```
`cargo run` / `cargo test` work out of the box because Cargo passes the
resolved library directory to the linker for you. A **directly executed** or
**deployed** binary, however, needs a runtime search path (`rpath`) pointing at
the shared library — Cargo does not add one automatically. Use the
[`zvec-rust-build`](https://crates.io/crates/zvec-rust-build) build-script
helper from your binary crate to emit the `rpath` and stage the library beside
the executable:
```toml
[dependencies]
zvec-rust = "0.8.0"
[build-dependencies]
zvec-rust-build = "0.8.0"
```
```rust
// build.rs
fn main() {
zvec_rust_build::configure();
}
```
Without the helper you must instead set `DYLD_LIBRARY_PATH` (macOS) /
`LD_LIBRARY_PATH` (Linux) at runtime.
### Option 2: Custom Build
If you want to build the zvec C library yourself (e.g., for a custom configuration or unsupported platform), set the `ZVEC_LIB_DIR` environment variable:
```bash
# Build zvec from source
git clone https://github.com/alibaba/zvec.git && cd zvec
mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_C_BINDINGS=ON
make -j$(nproc)
# Point zvec-rust to your custom build
export ZVEC_LIB_DIR=/path/to/zvec/build/lib
```
Or use the built-in Makefile for local development:
```bash
make setup # Install dev tools + init git submodule
make zvec-build # Build the zvec C library from submodule
make test-all # Run all tests
```
### Library Resolution Order
The build script resolves the C library in this order:
1. **`ZVEC_LIB_DIR`** environment variable (highest priority)
2. **Sibling checkout**: `../zvec/build/lib`
3. **Git submodule**: `vendor/zvec/build/lib`
4. **Vendor directory**: `vendor/lib/`
5. **Prebuilt download**: from GitHub Releases (automatic)
6. **Auto-build**: clone and build from source (requires `git`, `cmake`, C++17 compiler)
Set `ZVEC_AUTO_BUILD=0` to disable steps 5 and 6.
### Prebuilt Downloads and Failure Handling
Prebuilt SDKs remain cached at `$OUT_DIR/zvec-prebuilt`, under Cargo's `target`
directory. Cleaning the corresponding build output removes this cache.
Cache reuse requires successful extraction, the expected native library, and a
completion marker; an unmarked cache entry is downloaded again and replaced only
after the new download is verified, so a failed download never destroys the copy
you already have.
Failed curl downloads fall back to wget, then PowerShell on Windows. Command
failures report their exit status and diagnostics. Archives that fail extraction
or lack the expected library are retained at the printed path until the next
download attempt or clean.
If acquisition is enabled and every method fails, the build script stops with an
explicit error instead of deferring the failure to the linker. Acquisition is
enabled by the `bundled` feature unless `ZVEC_AUTO_BUILD=0` is set. When it is
disabled the build script only warns, so `cargo check`, `cargo doc` and
`cargo publish` keep working for setups that provide the library themselves,
such as a system `libzvec_c_api` reached through `ZVEC_LIB_DIR` or `RUSTFLAGS`.
## Quick Start
```rust
use zvec_rust::*;
fn main() -> zvec_rust::Result<()> {
// 1. Initialize the engine
initialize(None)?;
// 2. Define schema
let schema = CollectionSchema::builder("my_collection")
// Note 1: use `?` to unwrap the Result returned by FieldSchema::new
.add_field(FieldSchema::new("id", DataType::String, false, 0)?)
// Note 2: use `?` to unwrap the Result returned by IndexParams::hnsw
.add_vector_field(
"embedding",
DataType::VectorFp32,
128,
IndexParams::hnsw(MetricType::Cosine, 16, 200)?
)
.build()?;
println!("Schema built successfully.");
// 3. Create and open the collection (the directory will be created if it does not exist)
// "./data" is the local storage path
let collection = Collection::create_and_open("./data", &schema, None)?;
println!(" Collection opened.");
// 4. Insert data
let mut doc = Doc::new()?;
doc.set_pk("doc1");
doc.add_string("id", "doc1")?;
// Build a 128-dim vector filled with 0.1
let vec_data = vec![0.1_f32; 128];
doc.add_vector_f32("embedding", &vec_data)?;
// `insert` accepts a slice of &[&Doc]
collection.insert(&[&doc])?;
println!("Document inserted.");
// 5. Vector similarity search
// Query vector: filled with 0.2
let query_vec = vec![0.2_f32; 128];
let query = SearchQuery::new("embedding", &query_vec, 10)?;
let results = collection.query(&query)?;
println!("Search Results:");
for result in &results {
let pk = result.get_pk().unwrap_or("unknown");
let score = result.get_score();
println!(" PK: {}, Score: {:.4}", pk, score);
}
// 6. Shutdown the engine
shutdown()?;
println!("Test Finished!");
Ok(())
}
```
## Examples
Run any example with `cargo run --example <name>`:
| `basic` | End-to-end workflow: schema → insert → query → fetch → delete |
| `schema_builder` | Various schema configurations: field types, index types, quantization |
| `vector_search` | Vector query patterns: simple, builder, filter, output fields, HNSW params |
| `crud_operations` | Full CRUD: insert, fetch, update, upsert, delete, stats, flush |
| `config_logging` | Library configuration: memory limits, thread counts, logging |
```bash
# Using cargo directly (requires ZVEC_LIB_DIR / DYLD_LIBRARY_PATH)
export DYLD_LIBRARY_PATH=vendor/zvec/build/lib
cargo run --example basic
cargo run --example vector_search
```
## API Overview
### Initialization
| `initialize(config)` | Initialize the library (call once); pass `None` for defaults |
| `shutdown()` | Release all resources |
| `version()` | Get version string |
| `is_initialized()` | Check initialization status |
| `io_backend_type()` / `io_backend_type_name(code)` | Active DiskAnn I/O backend code and name (`io_uring`, `libaio`, `pread`, `windows_overlapped`) |
| `io_backend_description()` | Human-readable description of the active I/O backend |
Use [`ConfigBuilder`](zvec/src/config.rs) to customize memory limits, thread counts, and logging:
```rust
let config = ConfigBuilder::new()
.memory_limit(1024 * 1024 * 1024)
.num_threads(4)
.enable_console_log(true)
.build();
initialize(Some(&config))?;
```
#### jieba FTS tokenizer dictionary
`libzvec_c_api` embeds cppjieba's default dictionary and HMM model
(LZ4-compressed), so the `jieba` FTS tokenizer reads them from the library
itself — Chinese full-text search works with no dictionary files on disk and
no extra configuration:
```rust
initialize(None)?;
let schema = CollectionSchema::builder("articles")
.add_field(FieldSchema::new("id", DataType::String, false, 0)?)
.add_indexed_field("content", DataType::String,
IndexParams::fts(Some("jieba"), None, None)?)
.build()?;
```
To use a custom dictionary instead of the embedded default, set one of the
following (highest priority first); the directory must contain
`jieba.dict.utf8` and `hmm_model.utf8`:
- per-field `extra_params` JSON with `"jieba_dict_dir"` — per index
- `ZVEC_JIEBA_DICT_DIR` environment variable — process-wide, read when the tokenizer initializes
The `user_dict_path` extra param adds entries on top of whichever dictionary
was selected, including the embedded default.
### Schema Definition
```rust
let schema = CollectionSchema::builder("name")
.add_field(FieldSchema::new("field", DataType::String, false, 0)?)
.add_vector_field("vec", DataType::VectorFp32, 128,
IndexParams::hnsw(MetricType::Cosine, 16, 200)?)
.build()?;
```
### Collection Operations
| `Collection::create_and_open()` | Create a new collection |
| `Collection::open()` | Open an existing collection |
| `collection.insert(&docs)` | Insert documents |
| `collection.update(&docs)` | Update documents |
| `collection.upsert(&docs)` | Insert or update |
| `collection.delete(&pks)` | Delete by primary keys |
| `collection.delete_by_filter(filter)` | Delete documents matching a filter expression |
| `collection.query(&query)` | Scalar filtering, full-text, or vector similarity search |
| `collection.multi_query(&query)` | Multi-route search with RRF / weighted rerank |
| `collection.fetch(&pks)` | Fetch by primary keys |
| `collection.fetch_with_options(&pks, fields, include_vector)` | Fetch with output-field control |
| `collection.iter()` / `iter_with_options(fields, include_vector)` | Iterate over all documents (isolated snapshot) |
| `collection.create_index(field, params)` / `drop_index(field)` | Runtime index management |
| `collection.add_column(schema, default_expr)` / `drop_column(name)` / `alter_column(name, new_name, new_schema)` | Schema evolution (DDL) |
| `collection.optimize()` | Rebuild indexes / merge segments |
| `collection.stats()` | Get collection statistics |
| `collection.flush()` | Flush to disk |
**Caution:** a projected `fetch_with_options(pks, Some(&[..]), ..)` — or one with
`include_vector = false` — returns a document whose unprojected columns are
empty, not absent. Writing that document back with `upsert` / `update` overwrites
those columns: nullable ones silently become null, and a missing non-nullable one
is rejected with `InvalidArgument`. For a read-modify-write cycle, fetch the whole
document with `fetch_with_options(pks, None, true)`.
### Document Operations
```rust
let mut doc = Doc::new()?;
doc.set_pk("my_pk");
doc.add_string("name", "value")?;
doc.add_i64("count", 42)?;
doc.add_vector_f32("embedding", &[0.1, 0.2, 0.3])?;
// Getters return `Result<Option<T>>` — `?` only unwraps the Result.
// Use `unwrap_or_default()` / `expect(..)` etc. to handle the Option.
let name: Option<String> = doc.get_string("name")?;
let count: Option<i64> = doc.get_i64("count")?;
```
### Scalar Query
Use `SearchQuery::scalar(topk)` to filter scalar fields without a query vector,
FTS payload, or target field. It also works on collections with no vector fields:
```rust
let mut query = SearchQuery::scalar(10)?;
query.set_filter("relative_path = 'src/main.rs'")?;
query.set_output_fields(&["relative_path"])?;
let results = collection.query(&query)?;
```
`topk` limits the number of returned documents, and their order is unspecified.
A query without a filter returns up to `topk` documents; it does not enumerate
all matches when there are more than that limit.
`SearchQuery::builder()` produces the same query when neither `vector(..)` nor an
FTS clause is set; `field_name(..)` must then be omitted.
### Vector Query
```rust
// Simple query
let query = SearchQuery::new("embedding", &query_vec, 10)?;
// Builder pattern with filters
let query = SearchQuery::builder()
.field_name("embedding")
.vector(&query_vec)
.topk(10)
.filter("category = 'tech'")
.output_fields(&["id", "name"])
.build()?;
```
The builder infers the query kind from what was set: `vector(..)` builds a dense
vector query (adding an FTS clause makes it a hybrid search), an FTS clause
without a vector builds a keyword-only query, and neither builds a scalar query.
`SearchQuery::fts(field, fts, topk)` is the direct keyword-only constructor.
### Multi-Query (Hybrid Search)
`MultiQuery` combines multiple sub-queries (dense vector, sparse vector, or FTS) with RRF or weighted reranking:
```rust
// FTS + vector hybrid search with RRF reranking
let mut sub_vec = SubQuery::new()?;
sub_vec.set_field_name("embedding")?;
sub_vec.set_query_vector(&query_vec)?;
sub_vec.set_num_candidates(50)?;
let mut fts = Fts::new()?;
fts.set_match_string("Rust vector database")?;
let mut sub_fts = SubQuery::new()?;
sub_fts.set_field_name("content")?;
sub_fts.set_fts(&fts)?;
sub_fts.set_num_candidates(50)?;
let mut mq = MultiQuery::new()?;
mq.set_topk(10)?;
mq.set_rerank_rrf(60)?; // rank constant for RRF
mq.add_sub_query(&sub_vec)?;
mq.add_sub_query(&sub_fts)?;
let results = collection.multi_query(&mq)?;
// Weighted reranking
mq.set_rerank_weighted(&[0.7, 0.3])?; // weights per sub-query
```
## Supported Types
| **Scalar** | `Bool`, `Int32`, `Int64`, `Uint32`, `Uint64`, `Float`, `Double`, `String`, `Binary` |
| **Vector** | `VectorFp16`, `VectorFp32`, `VectorFp64`, `VectorInt4`, `VectorInt8`, `VectorInt16`, `VectorBinary32`, `VectorBinary64` |
| **Sparse** | `SparseVectorFp16`, `SparseVectorFp32` |
| **Array** | `ArrayBool`, `ArrayInt32`, `ArrayInt64`, `ArrayUint32`, `ArrayUint64`, `ArrayFloat`, `ArrayDouble`, `ArrayString`, `ArrayBinary` |
## Index Types
Available distance metrics: `L2`, `Ip`, `Cosine`, `MipsL2`.
| HNSW | `IndexParams::hnsw(metric, m, ef)` | Graph index (recommended) |
| HNSW+Q | `IndexParams::hnsw_with_quantize(...)` | HNSW with quantization |
| IVF | `IndexParams::ivf(metric, nlist, niters, soar)` | Inverted file index |
| IVF RaBitQ | `IndexParams::ivf_rabitq(metric, nlist, total_bits, sample_count)` | IVF with RaBitQ quantization |
| Flat | `IndexParams::flat(metric)` | Brute-force index |
| DiskANN | `IndexParams::diskann(metric, max_degree, list_size, pq_chunk_num)` | Disk-based graph index for large datasets (Linux x86_64/ARM64, macOS ARM64, Windows x86_64, 64-bit Android/iOS) |
| Invert | `IndexParams::invert(range, wildcard)` | Scalar field index |
| FTS | `IndexParams::fts(tokenizer, filters, extra)` | Full-text search index |
With `Int8` / `Int4` quantization, `params.set_quantizer_enable_rotate(true)` randomly rotates
vectors before quantization to reduce the quantization error. The rotation matrix is stored with
the index and applied to query vectors at search time.
## Testing
```bash
# Using Makefile (recommended — auto-detects library paths)
make test-unit # Unit tests (no C library required)
make test-integration # Integration tests (requires C library)
make test-all # All tests (unit + integration + doc)
# Using cargo directly (requires ZVEC_LIB_DIR / DYLD_LIBRARY_PATH)
cargo test --lib
cargo test --test integration_test
# Fuzz tests (requires nightly)
cargo install cargo-fuzz
cargo +nightly fuzz run fuzz_types -- -max_total_time=60
# Benchmarks
make bench
# Code coverage
cargo install cargo-llvm-cov
./scripts/coverage.sh --html
```
## Keeping in Sync with zvec Core
This SDK tracks the [zvec](https://github.com/alibaba/zvec) C-API. When the upstream C-API changes:
1. Update `zvec-sys/src/lib.rs` with new FFI declarations
2. Add safe wrappers in the `zvec` crate
3. Update integration tests to cover new functionality
4. Run the full test suite to verify compatibility
The CI pipeline automatically clones the latest zvec and builds the C library, ensuring FFI compatibility on every PR.
## Contributing
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Ensure all tests pass (`cargo test`)
4. Ensure code is formatted (`cargo fmt --all -- --check`)
5. Ensure clippy is clean (`cargo clippy --workspace --all-targets -- -D warnings`)
6. Submit a pull request
## License
Apache-2.0 — see [LICENSE](LICENSE).