Module transformers

Various transformers for chunking, embedding and transforming data

These transformers are generic over their implementation and many require a swiftide integration to be configured.

Transformers that prompt have a default prompt configured. Prompts can be customized and tailored, supporting Jinja style templating based on tera.

See swiftide_core::prompt::Prompt and swiftide_core::prompt::PromptTemplate

Re-exports

• pub use chunk_markdown::ChunkMarkdown;
• pub use chunk_text::ChunkText;
• pub use embed::Embed;
• pub use metadata_keywords::MetadataKeywords;
• pub use metadata_qa_text::MetadataQAText;
• pub use metadata_summary::MetadataSummary;
• pub use metadata_title::MetadataTitle;
• pub use sparse_embed::SparseEmbed;

Modules

• chunk_markdown
  Chunk markdown content into smaller pieces
• chunk_text
  Chunk text content into smaller pieces
• embed
  Generic embedding transformer
• metadata_keywords
  Extract keywords from a node and add them as metadata This module defines the MetadataKeywords struct and its associated methods, which are used for generating metadata in the form of keywords for a given text. It interacts with a client (e.g., OpenAI) to generate the keywords based on the text chunk in a Node.
• metadata_qa_text
  Generates questions and answers from a given text chunk and adds them as metadata. This module defines the MetadataQAText struct and its associated methods, which are used for generating metadata in the form of questions and answers from a given text. It interacts with a client (e.g., OpenAI) to generate these questions and answers based on the text chunk in an Node.
• metadata_summary
  Generate a summary and adds it as metadata This module defines the MetadataSummary struct and its associated methods, which are used for generating metadata in the form of a summary for a given text. It interacts with a client (e.g., OpenAI) to generate the summary based on the text chunk in an Node.
• metadata_title
  Generate a title and adds it as metadata This module defines the MetadataTitle struct and its associated methods, which are used for generating metadata in the form of a title for a given text. It interacts with a client (e.g., OpenAI) to generate these questions and answers based on the text chunk in an Node.
• sparse_embed
  Generic embedding transformer