Module transformers

Various transformers for chunking, embedding and transforming data

These transformers are generic over their implementation and many require a swiftide integration to be configured.

Transformers that prompt have a default prompt configured. Prompts can be customized and tailored, supporting Jinja style templating based on tera.

See swiftide_core::prompt::Prompt and swiftide_core::prompt::PromptTemplate

Re-exports

• pub use chunk_markdown::ChunkMarkdown;
• pub use chunk_text::ChunkText;
• pub use embed::Embed;
• pub use metadata_keywords::MetadataKeywords;
• pub use metadata_qa_text::MetadataQAText;
• pub use metadata_summary::MetadataSummary;
• pub use metadata_title::MetadataTitle;
• pub use sparse_embed::SparseEmbed;

Modules

• chunk_markdown
  Chunk markdown content into smaller pieces
• chunk_text
  Chunk text content into smaller pieces
• embed
  Generic embedding transformer
• metadata_keywords
  Extract keywords from a node and add them as metadata
• metadata_qa_text
  Generates questions and answers from a given text chunk and adds them as metadata.
• metadata_summary
  Generate a summary and adds it as metadata
• metadata_title
  Generate a title and adds it as metadata
• sparse_embed
  Generic embedding transformer