OpenViking-rs

An Agent-native context database library for Rust.

Overview

OpenViking-rs is a Rust library that provides a context database specifically designed for AI agents. It enables efficient storage, retrieval, and management of contextual information that AI agents need to perform their tasks effectively.

This is a Rust port of the OpenViking project, reimplemented in pure Rust for better performance, memory safety, and easier integration with other Rust projects.

Features

Virtual File System (VikingFS) - A virtual file system with Viking URI support (viking://[scope]/[path])
Three-Layer Context Storage - L0 (Abstract), L1 (Overview), L2 (Full Content)
Vector Database - HNSW-based approximate nearest neighbor search with scalar filtering
Session Management - Conversation tracking with memory extraction and deduplication
Hierarchical Retrieval - Directory-recursive retrieval strategy with intent analysis
File Parsing - Support for multiple file formats (text, markdown, code, PDF, Office documents)
Async Processing Queue - Background embedding and semantic processing
OVPack Import/Export - Package resources for sharing and backup

Installation

Add this to your Cargo.toml:

[dependencies]
openviking-rs = "0.1.0"

Feature Flags

pdf - Enable PDF parsing support
office - Enable Word/Excel/PowerPoint parsing support
html - Enable HTML parsing support
epub - Enable EPUB parsing support
full - Enable all optional features

[dependencies]
openviking-rs = { version = "0.1.0", features = ["full"] }

Quick Start

Implement Required Traits

OpenViking-rs uses trait abstractions for embedding and VLM models, allowing you to integrate any provider:

use openviking_rs::{Embedder, Vlm, Image, Result};
use async_trait::async_trait;

struct MyEmbedder;

#[async_trait]
impl Embedder for MyEmbedder {
    async fn embed(&self, text: &str) -> Result<Vec<f32>> {
        // Your embedding implementation
        Ok(vec![0.0; 768])
    }

    async fn embed_batch(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
        // Batch embedding for efficiency
        Ok(texts.iter().map(|_| vec![0.0; 768]).collect())
    }

    fn dimension(&self) -> usize {
        768
    }

    fn name(&self) -> &str {
        "my-embedder"
    }
}

struct MyVlm;

#[async_trait]
impl Vlm for MyVlm {
    async fn generate(&self, prompt: &str, images: &[Image]) -> Result<String> {
        // Your VLM implementation
        Ok("response".to_string())
    }

    async fn generate_stream(&self, prompt: &str, images: &[Image]) -> Result<impl Stream<Item = String>> {
        // Streaming generation
    }

    fn name(&self) -> &str {
        "my-vlm"
    }
}

Using the Client

use openviking_rs::{AsyncOpenViking, AsyncOpenVikingBuilder, VikingUri, Scope};

#[tokio::main]
async fn main() -> Result<()> {
    // Create client
    let mut client = AsyncOpenVikingBuilder::default()
        .storage_path("./data")
        .embedder(MyEmbedder)
        .vlm(MyVlm)
        .build()
        .await?;

    // Initialize
    client.initialize().await?;

    // Add a resource
    let uri = VikingUri::new(Scope::Resources, "documents/readme.md");
    client.add_resource(
        "/path/to/file.md",
        Some(&uri),
        "import",
        "process",
        true,
        None,
    ).await?;

    // Search
    let results = client.find(
        "search query",
        &VikingUri::new(Scope::Resources, ""),
        10,
        None,
    ).await?;

    for result in results.resources() {
        println!("{}: {:.2}", result.uri(), result.score());
    }

    // Close
    client.close().await?;

    Ok(())
}

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.

Acknowledgments

This project is a Rust implementation inspired by and ported from OpenViking.

We would like to express our sincere gratitude to:

Volcengine Team - The original creators of OpenViking for their innovative design of an Agent-native context database
OpenViking Contributors - All the developers who contributed to the original Python/C++ implementation
Open Source Community - For the excellent Rust crates that made this port possible

If you use this library, please consider starring the original OpenViking repository to show your support for the original project.

openviking-rs 0.1.0