Crate kalosm_language

Kalosm Language

This crate is a collection of language processing utilities for the Kalosm project.

There are two main traits in this crate: prelude::ModelExt for large language model (that implement prelude::Model) and prelude::Embedder for text embedding model.

Those two traits interact with the context that this crate provides. Many different types in this crates can be converted to a prelude::Document with the prelude::IntoDocument or prelude::IntoDocuments trait:

• prelude::Page: Handles scraping a webpage from a request of headless browser
• prelude::SearchQuery: Handles searching with a search engine and scaping the result
• prelude::CrawlingCallback: Handles crawling a set of webpages
• prelude::FsDocument: Handles reading a document from the file system
• prelude::DocumentFolder: Handles reading an entire folder of documents from the file system

Re-exports

• pub use kalosm_language_model;
• pub use kalosm_llama;
• pub use kalosm_sample;
• pub use rbert;
• pub use rphi;

Modules

• chat
  A chat interface that builds on top of kalosm_language_model::Model
• context
  Context for language models to consume.
• prelude
  A prelude of commonly used items in kalosm-language
• search
  The index module contains different types of search indexes that can be used to search for crate::context::Documents created from crate::context::IntoDocument or crate::context::IntoDocuments
• task
  A task interface that builds on top of kalosm_language_model::Model
• tool
  Tools that can be used by kalosm_language_model::Model’s to perform actions.
• vector_db
  A vector database that can be used to store embeddings and search for similar embeddings.