Expand description
This module serves as the main entry point for indexing in Swiftide.
The indexing system in Swiftide is designed to handle the asynchronous processing of large volumes of data, including loading, transforming, and storing data chunks.
Modules§
- __
mock_ Mock Batchable Transformer - __
mock_ Mock Batchable Transformer_ Batchable Transformer - __
mock_ Mock Batchable Transformer_ Clone - __
mock_ Mock Chunker Transformer - __
mock_ Mock Chunker Transformer_ Chunker Transformer - __
mock_ Mock Chunker Transformer_ Clone - __
mock_ Mock Embedding Model - __
mock_ Mock Embedding Model_ Clone - __
mock_ Mock Embedding Model_ Embedding Model - __
mock_ Mock Loader - __
mock_ Mock Loader_ Clone - __
mock_ Mock Loader_ Loader - __
mock_ Mock Node Cache - __
mock_ Mock Node Cache_ Clone - __
mock_ Mock Node Cache_ Node Cache - __
mock_ Mock Persist - __
mock_ Mock Persist_ Clone - __
mock_ Mock Persist_ Persist - __
mock_ Mock Simple Prompt - __
mock_ Mock Simple Prompt_ Clone - __
mock_ Mock Simple Prompt_ Simple Prompt - __
mock_ Mock Sparse Embedding Model - __
mock_ Mock Sparse Embedding Model_ Clone - __
mock_ Mock Sparse Embedding Model_ Sparse Embedding Model - __
mock_ Mock Transformer - __
mock_ Mock Transformer_ Clone - __
mock_ Mock Transformer_ Transformer - loaders
- The
loadersmodule provides functionality for loading files from a specified directory. It includes theFileLoaderstruct which is used to filter and stream files based on their extensions. - persist
- Storage implementations for persisting data
- transformers
Structs§
- Backoff
Configuration - Backoff configuration for api calls. Each time an api call fails backoff will wait an increasing period of time for each subsequent retry attempt. see https://docs.rs/backoff/latest/backoff/ for more details.
- Indexing
Defaults - Indexing
Defaults Inner - Indexing
Stream - An asynchronous stream of
Nodeitems. - Language
Model With Back Off - Metadata
- Mock
Batchable Transformer - Mock
Chunker Transformer - Mock
Embedding Model - Mock
Loader - Mock
Node Cache - Mock
Persist - Mock
Simple Prompt - Mock
Sparse Embedding Model - Mock
Transformer - Node
- Represents a unit of data in the indexing process.
- Node
Builder - Builder for
Node. - Pipeline
- A pipeline for indexing files, adding metadata, chunking, transforming, embedding, and then storing them.
Enums§
- Embed
Mode - Embed mode of the pipeline.
- Embedded
Field - Type of Embeddable stored in model.
Traits§
- Batchable
Transformer - Transforms batched single nodes into streams of nodes
- Chunker
Transformer - Turns one node into many nodes
- DynClone
- This trait is implemented by any type that implements [
std::clone::Clone]. - Embedding
Model - Embeds a list of strings and returns its embeddings. Assumes the strings will be moved.
- Loader
- Starting point of a stream
- Node
Cache - Caches nodes, typically by their path and hash Recommended to namespace on the storage
- Persist
- Persists nodes
- Simple
Prompt - Given a string prompt, queries an LLM
- Sparse
Embedding Model - Embeds a list of strings and returns its embeddings. Assumes the strings will be moved.
- Transformer
- Transforms single nodes into single nodes
- With
Batch Indexing Defaults - Allows for passing defaults from the pipeline to the batch transformer Required for batch transformers as at least a marker, implementation is not required
- With
Indexing Defaults - Allows for passing defaults from the pipeline to the transformer Required for batch transformers as at least a marker, implementation is not required