Skip to main content

lvv/
lib.rs

1#![warn(missing_docs)]
2#![cfg_attr(docsrs, feature(doc_cfg))]
3
4//! Build pipelines that turn structured datasets into vector embeddings.
5//!
6//! `lvv` provides the pieces needed to load records, generate embeddings with
7//! Ollama or OpenAI, cache those embeddings, and write the result to Qdrant or
8//! another [`db::Sink`]. Jobs can be collected in a [`jobs::job_queue::JobQueue`]
9//! and sent to more than one sink in registration order.
10//!
11//! # Pipeline
12//!
13//! 1. Create an [`intake::dataset::DataSet`] directly or read records through
14//!    an [`intake::source::Source`].
15//! 2. Describe the embedding request with [`jobs::job::JobBuilder`].
16//! 3. Add one or more [`db::Sink`] implementations to a
17//!    [`jobs::job_queue::JobQueue`] and call its `run` method.
18//!
19//! # Typed records
20//!
21//! Records that already exist as Rust types can describe their own points.
22//! Implement [`points::VectorDatabaseItem`], or derive it with the `derive`
23//! feature, and each value becomes a [`points::VectorPointDraft`]: a category,
24//! the text to embed, and the payload to store with the vector. Embed the
25//! descriptions with [`inference::EmbeddingProvider::embed_texts`]. See
26//! [`points`].
27//!
28//! # Transforming records with LLMs
29//!
30//! Records can be rewritten by a chat model before they are embedded: to
31//! summarize long text, extract keywords, translate, or clean up inconsistent
32//! fields. A [`transform::Transform`] holds the instructions, what to send for
33//! each record and where the reply goes; [`transform::Llm::run`] applies it to
34//! a slice of records against Ollama or any OpenAI-compatible server. Replies
35//! can be plain text or typed JSON, checked against a schema.
36//!
37//! Every record gets an outcome in the returned [`transform::Report`], in
38//! record order. Rate limits and server errors are retried, a wrong model name
39//! stops the run after a few requests, and with a cache an interrupted run
40//! resumes where it stopped. With the `derive` feature, marking the field that
41//! holds the reply `#[lvv(description)]` makes it part of the embedded text.
42//! See [`transform`].
43//!
44//! # Backends and environment variables
45//!
46//! Models are reached through the [`backend`] clients. Ollama uses
47//! `OLLAMA_URL`, defaulting to `http://127.0.0.1:11434`. OpenAI reads
48//! `OPENAI_API_KEY` from the environment, or from a `.env` file when the
49//! variable isn't set. Remote Qdrant configuration reads `QDRANT_API_KEY`
50//! after loading a `.env` file, and fails if there isn't one.
51//!
52//! # Cargo features
53//!
54//! None are enabled by default.
55//!
56//! | Feature    | Enables |
57//! |------------|---------|
58//! | `derive`   | `#[derive(VectorDatabaseItem)]` and `#[derive(VectorDatabase)]` from [`lvv-macros`](https://docs.rs/lvv-macros), re-exported in [`points`] |
59//! | `postgres` | `intake::PostgresSource` and `db::PostgresSink` |
60//! | `sql`      | `intake::SqlSource` for SQLite and MySQL |
61//! | `http`     | `intake::HttpSource` for paginated JSON APIs |
62//!
63//! # Example
64//!
65//! ```no_run
66//! use lvv::{
67//!     db::{Distance, QdrantSink},
68//!     db::vector_database::{DatabaseParams, Location},
69//!     intake::dataset::DataSet,
70//!     jobs::{JobBuilder, Provider},
71//!     jobs::job_queue::JobQueue,
72//! };
73//! use std::sync::Arc;
74//!
75//! # async fn example() -> anyhow::Result<()> {
76//! let dataset = DataSet::new("articles.json", "articles", vec![
77//!     serde_json::json!({"title": "Rust documentation"}),
78//! ]);
79//! let job = JobBuilder::default()
80//!     .dataset(dataset)
81//!     .provider(Provider::Ollama("nomic-embed-text".into()))
82//!     .dims(768_u64)
83//!     .extends(false)
84//!     .distance(Distance::Cosine)
85//!     .collection_name()
86//!     .build()?;
87//!
88//! let params = DatabaseParams::new(
89//!     Location::new_local("http://localhost:6334"),
90//!     "articles".into(),
91//!     Distance::Cosine,
92//!     768,
93//! );
94//! let mut queue = JobQueue::from_vec(vec![job]);
95//! queue.with_sink(Arc::new(QdrantSink::new(params)));
96//! queue.run().await?;
97//! # Ok(())
98//! # }
99//! ```
100//!
101//! # Full example
102//!
103//! [`examples/full_pipeline.rs`](https://github.com/egonik-unlp/lvv/blob/main/examples/full_pipeline.rs)
104//! uses every stage in one program, with the `derive` feature:
105//!
106//! 1. [`intake::FileSource`] loads records from JSON Lines, CSV and JSON files
107//!    into structs that derive `VectorDatabaseItem`.
108//! 2. A [`transform::Transform`] run by [`transform::Llm`] writes a summary into
109//!    each position, in a field marked `#[lvv(description)]`, caching the
110//!    summaries between runs.
111//! 3. `#[derive(VectorDatabase)]` turns the records into points.
112//! 4. [`inference::EmbeddingProvider::embed_texts`] embeds each category's descriptions,
113//!    reusing vectors from a [`cache::cache_embeddings::Cache`].
114//! 5. Each category becomes a [`jobs::job::Job`] with precomputed embeddings.
115//! 6. A [`jobs::job_queue::JobQueue`] writes the jobs to Qdrant.
116//!
117//! ```text
118//! cargo run --example full_pipeline --features derive
119//! QDRANT_URL=http://localhost:6334 cargo run --example full_pipeline --features derive
120//! ```
121
122/// HTTP clients for chat and embedding models.
123pub mod backend;
124/// Reusable embedding cache support.
125pub mod cache;
126/// Vector database connections and pipeline sinks.
127pub mod db;
128/// Embedding clients.
129pub mod inference;
130/// Dataset types and source connectors.
131pub mod intake;
132/// Embedding jobs and queue execution.
133pub mod jobs;
134/// Typed records as vector points, with optional derive macros.
135pub mod points;
136/// Rewriting records with a chat model before they are embedded.
137pub mod transform;
138
139/// Re-exports for code generated by `lvv-macros`, so users don't need these as
140/// direct dependencies. Not public API.
141#[doc(hidden)]
142pub mod __private {
143    pub use qdrant_client::Payload;
144    pub use serde;
145    pub use serde_json;
146}