Expand description
Build pipelines that turn structured datasets into vector embeddings.
lvv provides the pieces needed to load records, generate embeddings with
Ollama or OpenAI, cache those embeddings, and write the result to Qdrant or
another db::Sink. Jobs can be collected in a jobs::job_queue::JobQueue
and sent to more than one sink in registration order.
§Pipeline
- Create an
intake::dataset::DataSetdirectly or read records through anintake::source::Source. - Describe the embedding request with
jobs::job::JobBuilder. - Add one or more
db::Sinkimplementations to ajobs::job_queue::JobQueueand call itsrunmethod.
§Typed records
Records that already exist as Rust types can describe their own points.
Implement points::VectorDatabaseItem, or derive it with the derive
feature, and each value becomes a points::VectorPointDraft: a category,
the text to embed, and the payload to store with the vector. Embed the
descriptions with inference::EmbeddingProvider::embed_texts. See
points.
§Transforming records with LLMs
Records can be rewritten by a chat model before they are embedded: to
summarize long text, extract keywords, translate, or clean up inconsistent
fields. A transform::Transform holds the instructions, what to send for
each record and where the reply goes; transform::Llm::run applies it to
a slice of records against Ollama or any OpenAI-compatible server. Replies
can be plain text or typed JSON, checked against a schema.
Every record gets an outcome in the returned transform::Report, in
record order. Rate limits and server errors are retried, a wrong model name
stops the run after a few requests, and with a cache an interrupted run
resumes where it stopped. With the derive feature, marking the field that
holds the reply #[lvv(description)] makes it part of the embedded text.
See transform.
§Backends and environment variables
Models are reached through the backend clients. Ollama uses
OLLAMA_URL, defaulting to http://127.0.0.1:11434. OpenAI reads
OPENAI_API_KEY from the environment, or from a .env file when the
variable isn’t set. Remote Qdrant configuration reads QDRANT_API_KEY
after loading a .env file, and fails if there isn’t one.
§Cargo features
None are enabled by default.
| Feature | Enables |
|---|---|
derive | #[derive(VectorDatabaseItem)] and #[derive(VectorDatabase)] from lvv-macros, re-exported in points |
postgres | intake::PostgresSource and db::PostgresSink |
sql | intake::SqlSource for SQLite and MySQL |
http | intake::HttpSource for paginated JSON APIs |
§Example
use lvv::{
db::{Distance, QdrantSink},
db::vector_database::{DatabaseParams, Location},
intake::dataset::DataSet,
jobs::{JobBuilder, Provider},
jobs::job_queue::JobQueue,
};
use std::sync::Arc;
let dataset = DataSet::new("articles.json", "articles", vec![
serde_json::json!({"title": "Rust documentation"}),
]);
let job = JobBuilder::default()
.dataset(dataset)
.provider(Provider::Ollama("nomic-embed-text".into()))
.dims(768_u64)
.extends(false)
.distance(Distance::Cosine)
.collection_name()
.build()?;
let params = DatabaseParams::new(
Location::new_local("http://localhost:6334"),
"articles".into(),
Distance::Cosine,
768,
);
let mut queue = JobQueue::from_vec(vec![job]);
queue.with_sink(Arc::new(QdrantSink::new(params)));
queue.run().await?;§Full example
examples/full_pipeline.rs
uses every stage in one program, with the derive feature:
intake::FileSourceloads records from JSON Lines, CSV and JSON files into structs that deriveVectorDatabaseItem.- A
transform::Transformrun bytransform::Llmwrites a summary into each position, in a field marked#[lvv(description)], caching the summaries between runs. #[derive(VectorDatabase)]turns the records into points.inference::EmbeddingProvider::embed_textsembeds each category’s descriptions, reusing vectors from acache::cache_embeddings::Cache.- Each category becomes a
jobs::job::Jobwith precomputed embeddings. - A
jobs::job_queue::JobQueuewrites the jobs to Qdrant.
cargo run --example full_pipeline --features derive
QDRANT_URL=http://localhost:6334 cargo run --example full_pipeline --features deriveModules§
- backend
- HTTP clients for chat and embedding models. HTTP clients for chat and embedding models.
- cache
- Reusable embedding cache support. Persistent caching for generated embeddings.
- db
- Vector database connections and pipeline sinks. Persistence backends for embeddings and their source records.
- inference
- Embedding clients. Embedding clients.
- intake
- Dataset types and source connectors. Dataset containers and connectors for loading source records.
- jobs
- Embedding jobs and queue execution. Job descriptions and sequential pipeline execution.
- points
- Typed records as vector points, with optional derive macros. Turn typed records into vector points.
- transform
- Rewriting records with a chat model before they are embedded. Rewrite records with a chat model before they are embedded: summaries, keywords, translations, structured extraction.