Crate memista 

Memista: High-performance vector search service

Memista is a high-performance vector search library that combines SQLite for metadata storage with USearch for efficient vector similarity search. It provides both a library interface for embedding in Rust applications and a standalone HTTP server.

Features

• Fast Vector Similarity Search: Utilizes USearch for high-performance similarity search
• Persistent Storage: Stores text chunks and metadata in SQLite for durability
• Multi-Database Support: Supports multiple isolated databases through database_id partitioning
• Comprehensive API Documentation: Auto-generated OpenAPI documentation with Swagger, Redoc, and RapiDoc interfaces
• Environment-Based Configuration: Easily configurable through environment variables
• Asynchronous I/O: Built with async I/O for high performance and concurrency
• Memory Efficient: Uses optimized data structures for efficient memory usage

Library Usage

use memista::{AppState, Config, create_app};
use async_sqlite::{PoolBuilder, JournalMode};
use actix_web::HttpServer;
use std::sync::Arc;

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    // Load configuration
    let config = Config::from_env().expect("Failed to load configuration");
     
    // Create a database pool
    let db_pool = PoolBuilder::new()
        .path(&config.database_path)
        .journal_mode(JournalMode::Wal)
        .open()
        .await
        .expect("Failed to create database pool");

    // Create application state
    let app_state = Arc::new(AppState { db_pool });

    // Start the HTTP server
    let bind_address = format!("{}:{}", config.server_host, config.server_port);
    HttpServer::new(move || {
        create_app(app_state.clone())
    })
    .bind(bind_address)?
    .run()
    .await
}

HTTP API

Memista provides a RESTful HTTP API for vector search operations:

POST /v1/insert

Insert text chunks with their embeddings into a specified database.

curl -X POST http://localhost:8083/v1/insert 
  -H "Content-Type: application/json" 
  -d '{
    "database_id": "my_db",
    "chunks": [{
      "embedding": [0.1, 0.2],
      "text": "Sample text",
      "metadata": "{"source": "document1"}"
    }]
  }'

POST /v1/search

Search for similar chunks using vector embeddings.

curl -X POST http://localhost:8083/v1/search 
  -H "Content-Type: application/json" 
  -d '{
    "database_id": "my_db",
    "embeddings": [[0.1, 0.2]],
    "num_results": 5
  }'

DELETE /v1/drop

Drop a specific database and its associated vector index.

curl -X DELETE http://localhost:8083/v1/drop 
  -H "Content-Type: application/json" 
  -d '{
    "database_id": "my_db"
  }'

Structs

AppState

Application state shared across HTTP handlers

ChunkData

Represents a chunk of text with its vector embedding and metadata

Config

Configuration structure for the application

DropTableRequest

Request structure for dropping a database table and its index

InsertChunkRequest

Request structure for inserting chunks into the database

SearchRequest

Request structure for searching similar chunks

SearchResult

Structure representing a search result

Functions

create_app

Creates and configures the HTTP server

drop_table

API endpoint for dropping a database table and its associated vector index

ensure_table_exists

Ensures that a database table exists for the given database ID

insert_chunk

API endpoint for inserting chunks into the database

load_or_create_index

Loads an existing vector search index or creates a new one if it doesn’t exist

search

API endpoint for searching similar chunks using vector embeddings