archiver 0.3.0

RAG implementation with langchain-rust
archiver-0.3.0 is not a library.

Archiver is a command-line application designed to interact with large language models (LLMs) for testing purposes. It leverages Ollama for model integration and SurrealDB as the vector store. The application uses Langchain-rust as the framework for interacting with the Ollama and SurrealDB.

Change Log

v0.3.0

  • Index Directory: Add Directory Indexing for bulk Markdown file importing

v0.2.0

  • Prompt: Add prompt identifier and colored response from LLM
  • Error Handling: Better error handling with SurrealDB and Ollama
  • Chuncking: Added document chunking when adding files to SurrealDB
  • Mode Switch: Switched LLM chain to a Conversational Retriever from Conversational, making the LLM more efficient with responses from the VectorDB

v0.1.0

  • Conversational Memory: Maintain context across interactions with the LLM.
  • Vector Stores: Efficient storage and retrieval of high-dimensional vectors through SurrealDB.
  • Ollama Integration: Easy setup and use of Ollama for model management.
  • Indexing Files: Add Markdown files to SurrealDB.

Prerequisites

Before setting up Archiver, ensure you have the following installed on your system:

  • Rust (latest stable version)
  • Ollama
  • SurrealDB
  • just (command runner)

Install Archiver

  1. Install using Cargo: 
    cargo install archiver
    Make sure you have SurrealDB setup and Ollama before running Archiver

Setup

Ollama

  1. Install Ollama: 
    brew install ollama

Just

  1. Install Just: 
    cargo install just

SurrealDB

  1. Download and Install SurrealDB:

    curl -sSf https://install.surrealdb.com | sh

  2. Run SurrealDB:

    just db_dev

Setting up SurrealDB

Once Surreal is up and running you can use https://surrealist.app/ to access the instance of Surreal. But before doing that a Namespace and Table need to be setup. Fastest way to get this done is to run Archiver once.

  1. Set Namespace and Table:

    just run

  2. Add Markdown File to the SurrealDB:

    cargo run -- -a <markdownfile>

Archiver

  1. Setup from Source: 
    git clone https://github.com/yourusername/archiver.git
cd archiver
just db_dev # run in a seperate terminal
just run

License

This project is licensed under the Apache Livense (Version 2.0). See the LICENSE file for details.