Crate swiftide

Swiftide - Document and code indexation for retrieval augmented generation

Swiftide is a straightforward, easy-to-use, easy-to-extend asynchronous file ingestion and processing system. It is designed to be used in a RAG (Retrieval Augmented Generation) system. It is built to be fast and efficient, with a focus on parallel processing and asynchronous operations.

Part of the bosun.ai project. An upcoming platform for autonomous code improvement.

We <3 feedback: project ideas, suggestions, and complaints are very welcome. Feel free to open an issue.

Feature flags

Swiftide has little features enabled by default as there are some dependency heavy integrations.

Either use the ‘all’ feature flag (not recommended), or enable the integrations that you need. Each integration has a similarly named feature flag.

Re-exports

• pub use traits::*;
• pub use type_aliases::*;

Modules

• ingestion
  This module serves as the main entry point for the ingestion components in the Swiftide project. It re-exports the essential structs and types from the ingestion_node, ingestion_pipeline, and ingestion_stream modules, providing a unified interface for the ingestion functionality.
• integrations
  Integrations with various platforms and external services.
• loaders
  The loaders module provides functionality for loading files from a specified directory. It includes the FileLoader struct which is used to filter and stream files based on their extensions.
• persist
  Storage implementations for persisting data
• traits
  Traits in Swiftide allow for easy extendability
• transformers
  Various transformers for chunking, embedding and transforming data
• type_aliases