Crate swiftide_indexing 

Streaming indexing pipelines for Swiftide.

This crate provides the indexing Pipeline, the loaders::FileLoader, built-in transformers (chunking, embedding, LLM generated metadata) and the in-memory persist::MemoryStorage. The traits these build on live in swiftide-core; storage, models and code-aware transformers come from swiftide-integrations.

The swiftide facade re-exports this crate together with the core indexing types as swiftide::indexing. Prefer that unless you want a minimal dependency.

Guide: https://swiftide.rs/indexing/

use swiftide_indexing::{
    Pipeline, loaders::FileLoader, persist::MemoryStorage, transformers::ChunkMarkdown,
};

Pipeline::from_loader(FileLoader::new("README.md").with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(20..2048))
    .then_store_with(MemoryStorage::default())
    .run()
    .await?;

Modules

loaders

The loaders module provides functionality for loading files from a specified directory. It includes the FileLoader struct which is used to filter and stream files based on their extensions.

persist

Storage implementations for persisting data

transformers

Various transformers for chunking, embedding and transforming data

Structs

Pipeline

A pipeline for indexing files, adding metadata, chunking, transforming, embedding, and then storing them.