Module transformers

Various transformers for chunking, embedding and transforming data

These transformers are generic over their implementation and many require a swiftide integration to be configured.

Transformers that prompt have a default prompt configured. Prompts can be customized and tailored, supporting Jinja style templating based on tera.

See swiftide_core::prompt::Prompt and [swiftide_core::template::Template]

Re-exports

pub use chunk_markdown::ChunkMarkdown;

pub use chunk_text::ChunkText;

pub use embed::Embed;

pub use metadata_keywords::MetadataKeywords;

pub use metadata_qa_text::MetadataQAText;

pub use metadata_summary::MetadataSummary;

pub use metadata_title::MetadataTitle;

pub use sparse_embed::SparseEmbed;

Modules

chunk_markdown

Chunk markdown content into smaller pieces

chunk_text

Chunk text content into smaller pieces

embed

Generic embedding transformer

metadata_keywords

Extract keywords from a node and add them as metadata This module defines the MetadataKeywords struct and its associated methods, which are used for generating metadata in the form of keywords for a given text. It interacts with a client (e.g., OpenAI) to generate the keywords based on the text chunk in a Node.

metadata_qa_text

Generates questions and answers from a given text chunk and adds them as metadata. This module defines the MetadataQAText struct and its associated methods, which are used for generating metadata in the form of questions and answers from a given text. It interacts with a client (e.g., OpenAI) to generate these questions and answers based on the text chunk in an Node.

metadata_summary

Generate a summary and adds it as metadata This module defines the MetadataSummary struct and its associated methods, which are used for generating metadata in the form of a summary for a given text. It interacts with a client (e.g., OpenAI) to generate the summary based on the text chunk in an Node.

metadata_title

Generate a title and adds it as metadata This module defines the MetadataTitle struct and its associated methods, which are used for generating metadata in the form of a title for a given text. It interacts with a client (e.g., OpenAI) to generate these questions and answers based on the text chunk in an Node.

sparse_embed

Generic embedding transformer