Skip to main content

Crate lvv

Crate lvv 

Source
Expand description

Build pipelines that turn structured datasets into vector embeddings.

lvv provides the pieces needed to load records, generate embeddings with Ollama or OpenAI, cache those embeddings, and write the result to Qdrant or another db::Sink. Jobs can be collected in a jobs::job_queue::JobQueue and sent to more than one sink in registration order.

§Pipeline

  1. Create an intake::dataset::DataSet directly or read records through an intake::source::Source.
  2. Describe the embedding request with jobs::job::JobBuilder.
  3. Add one or more db::Sink implementations to a jobs::job_queue::JobQueue and call its run method.

§Typed records

Records that already exist as Rust types can describe their own points. Implement points::VectorDatabaseItem, or derive it with the derive feature, and each value becomes a points::VectorPointDraft: a category, the text to embed, and the payload to store with the vector. Embed the descriptions with inference::EmbeddingProvider::embed_texts. See points.

§Transforming records with LLMs

Records can be rewritten by a chat model before they are embedded: to summarize long text, extract keywords, translate, or clean up inconsistent fields. A transform::Transform holds the instructions, what to send for each record and where the reply goes; transform::Llm::run applies it to a slice of records against Ollama or any OpenAI-compatible server. Replies can be plain text or typed JSON, checked against a schema.

Every record gets an outcome in the returned transform::Report, in record order. Rate limits and server errors are retried, a wrong model name stops the run after a few requests, and with a cache an interrupted run resumes where it stopped. With the derive feature, marking the field that holds the reply #[lvv(description)] makes it part of the embedded text. See transform.

§Backends and environment variables

Models are reached through the backend clients. Ollama uses OLLAMA_URL, defaulting to http://127.0.0.1:11434. OpenAI reads OPENAI_API_KEY from the environment, or from a .env file when the variable isn’t set. Remote Qdrant configuration reads QDRANT_API_KEY after loading a .env file, and fails if there isn’t one.

§Cargo features

None are enabled by default.

FeatureEnables
derive#[derive(VectorDatabaseItem)] and #[derive(VectorDatabase)] from lvv-macros, re-exported in points
postgresintake::PostgresSource and db::PostgresSink
sqlintake::SqlSource for SQLite and MySQL
httpintake::HttpSource for paginated JSON APIs

§Example

use lvv::{
    db::{Distance, QdrantSink},
    db::vector_database::{DatabaseParams, Location},
    intake::dataset::DataSet,
    jobs::{JobBuilder, Provider},
    jobs::job_queue::JobQueue,
};
use std::sync::Arc;

let dataset = DataSet::new("articles.json", "articles", vec![
    serde_json::json!({"title": "Rust documentation"}),
]);
let job = JobBuilder::default()
    .dataset(dataset)
    .provider(Provider::Ollama("nomic-embed-text".into()))
    .dims(768_u64)
    .extends(false)
    .distance(Distance::Cosine)
    .collection_name()
    .build()?;

let params = DatabaseParams::new(
    Location::new_local("http://localhost:6334"),
    "articles".into(),
    Distance::Cosine,
    768,
);
let mut queue = JobQueue::from_vec(vec![job]);
queue.with_sink(Arc::new(QdrantSink::new(params)));
queue.run().await?;

§Full example

examples/full_pipeline.rs uses every stage in one program, with the derive feature:

  1. intake::FileSource loads records from JSON Lines, CSV and JSON files into structs that derive VectorDatabaseItem.
  2. A transform::Transform run by transform::Llm writes a summary into each position, in a field marked #[lvv(description)], caching the summaries between runs.
  3. #[derive(VectorDatabase)] turns the records into points.
  4. inference::EmbeddingProvider::embed_texts embeds each category’s descriptions, reusing vectors from a cache::cache_embeddings::Cache.
  5. Each category becomes a jobs::job::Job with precomputed embeddings.
  6. A jobs::job_queue::JobQueue writes the jobs to Qdrant.
cargo run --example full_pipeline --features derive
QDRANT_URL=http://localhost:6334 cargo run --example full_pipeline --features derive

Modules§

backend
HTTP clients for chat and embedding models. HTTP clients for chat and embedding models.
cache
Reusable embedding cache support. Persistent caching for generated embeddings.
db
Vector database connections and pipeline sinks. Persistence backends for embeddings and their source records.
inference
Embedding clients. Embedding clients.
intake
Dataset types and source connectors. Dataset containers and connectors for loading source records.
jobs
Embedding jobs and queue execution. Job descriptions and sequential pipeline execution.
points
Typed records as vector points, with optional derive macros. Turn typed records into vector points.
transform
Rewriting records with a chat model before they are embedded. Rewrite records with a chat model before they are embedded: summaries, keywords, translations, structured extraction.