Module swiftide_indexing::transformers

Various transformers for chunking, embedding and transforming data

These transformers are generic over their implementation and many require an [swiftide::integrations] to be configured.

Transformers that prompt have a default prompt configured. Prompts can be customized and tailored, supporting Jinja style templating based on [tera]. See [swiftide::prompt::Prompt] and [swiftide::prompt::PromptTemplate]

Re-exports

• pub use chunk_markdown::ChunkMarkdown;
• pub use embed::Embed;
• pub use metadata_keywords::MetadataKeywords;
• pub use metadata_qa_text::MetadataQAText;
• pub use metadata_summary::MetadataSummary;
• pub use metadata_title::MetadataTitle;
• pub use sparse_embed::SparseEmbed;

Modules

• chunk_markdown
  Chunk markdown content into smaller pieces
• embed
  Generic embedding transformer
• metadata_keywords
  Extract keywords from a node and add them as metadata
• metadata_qa_text
  Generates questions and answers from a given text chunk and adds them as metadata.
• metadata_summary
  Generate a summary and adds it as metadata
• metadata_title
  Generate a title and adds it as metadata
• sparse_embed
  Generic embedding transformer