rag_toolchain/
lib.rs

1//! # What is rag-toolchain ?
2//! this library aims to be a fast, rust native solution to integrating with GenAI
3//! and abstracting common workflows in a way the can be extended by consumers of the library.
4//! Theres a focus on concrete error types and a simple safe API for building out reliable
5//! programs that use GenAI. This library is still in its infancy, however, the intention is to
6//! grow out integrations and hopefully build some fun and cool stuff. There is some examples
7//! in the [GitHub](https://github.com/JackMatthewRimmer/rust-rag-toolchain) repository and also
8//! throughout the docs. I am open to contributors and suggestions so please get involved in the
9//! project if you wish.
10
11//! # Features:
12//! In this library all features are enabled by default. This is so you can install the library
13//! and have a play around. However, once you have landed on the features you which to be using
14//! make sure to disable "default-features" in you Cargo.toml and select only the ones you are
15//! using in order to keep binary size down and reduce compilation times.
16
17/// # Chains
18///
19/// The chains module represents a set of abstractions for common GenAI workflows
20/// (which would have been a better name !). Examples of this can be RAG or interacting
21/// with an LLM where the chat history is saved and resent with each request. This module
22/// is essentially the core of what the library is trying to provide and the rest of the modules
23/// are just tools in order to support these abstracted workflows.
24pub mod chains;
25
26/// # Chunkers
27///
28/// This module is aimed and the process of "chunking" or also referred to as "text splitting".
29/// Wondering what that is ? put shortly its a set of methods in order to break down text into
30/// smaller chunks / pieces. This is done to either keep embedded text short and consise or also
31/// to ensure that what your trying to embed isnt larger than the token limit the embedding mode
32/// has set out.
33pub mod chunkers;
34
35/// # Clients
36///
37/// The clients module is simply a place for all our implemented clients and there supporting code.
38/// This is going to be common for any embedding model's and LLM's we support and the easiest is the
39/// example of our OpenAI client.
40pub mod clients;
41
42/// # Common
43///
44/// The common module is a home for all our code that is shared across multiple modules. A prime example
45/// of this would be any domain specific types that can appear across the library such as the [`common::Chunk`] type.
46pub mod common;
47
48/// # Loaders
49///
50/// The aim of this module is to provide some easy data integrations for you AI workflows. This could be as simple as
51/// loading PDF's to potentially some third party services like notion. This has been less of a priority of the library
52/// so far but hopefully in the future we can start to build this aspect out.
53pub mod loaders;
54
55/// # Retrievers
56///
57/// Now what are retrievers ? retrievers are a bridge between an embedding model and a vector database allowing you to query
58/// with text and get text back without having to do the embedding of the query text manually.
59pub mod retrievers;
60
61/// # Stores
62///
63/// Stores are essentially just a reprisentation of a vector database allowing you to store any text into the vector database.
64/// Like retrievers
65pub mod stores;