llm-toolkit

Basic llm tools for rust

Motivation & Philosophy

High-level LLM frameworks like LangChain, while powerful, can be problematic in Rust. Their heavy abstractions and complex type systems often conflict with Rust's strengths, imposing significant constraints and learning curves on developers.

There is a clear need for a different kind of tool: a low-level, unopinionated, and minimalist toolkit that provides robust "last mile" utilities for LLM integration, much like how candle provides core building blocks for ML without dictating the entire application architecture.

This document proposes the creation of llm-toolkit, a new library crate designed to be the professional's choice for building reliable, high-performance LLM-powered applications in Rust.

Core Design Principles

1. Minimalist & Unopinionated: The toolkit will NOT impose any specific application architecture. Developers are free to design their own UseCases and Services. llm-toolkit simply provides a set of sharp, reliable "tools" to be called when needed.

2. Focused on the "Last Mile Problem": The toolkit focuses on solving the most common and frustrating problems that occur at the boundary between a strongly-typed Rust application and the unstructured, often unpredictable string-based responses from LLM APIs.

3. Minimal Dependencies: The toolkit will have minimal dependencies (primarily serde and minijinja) to ensure it can be added to any Rust project with negligible overhead and maximum compatibility.

Features

Prompt Generation

llm-toolkit offers two powerful and convenient ways to generate prompts, powered by the minijinja templating engine.

1. Ad-hoc Prompts with prompt! macro

For quick prototyping and flexible prompt creation, the prompt! macro provides a println!-like experience. You can pass any serde::Serialize-able data as context.

use llm_toolkit::prompt::prompt;
use serde::Serialize;

#[derive(Serialize)]
struct User {
    name: &'static str,
    role: &'static str,
}

let user = User { name: "Mai", role: "UX Engineer" };
let task = "designing a new macro";

let p = prompt!(
    "User {{user.name}} ({{user.role}}) is currently {{task}}.",
    user = user,
    task = task
).unwrap();

assert_eq!(p, "User Mai (UX Engineer) is currently designing a new macro.");

2. Structured Prompts with #[derive(ToPrompt)]

For core application logic, you can derive the ToPrompt trait on your structs to generate prompts in a type-safe way.

Setup:

First, enable the derive feature in your Cargo.toml:

[dependencies]
llm-toolkit = { version = "0.1.0", features = ["derive"] }
serde = { version = "1.0", features = ["derive"] }

Usage:

Then, use the #[derive(ToPrompt)] and #[prompt(...)] attributes on your struct. The struct must also derive serde::Serialize.

use llm_toolkit::ToPrompt;
use serde::Serialize;

#[derive(ToPrompt, Serialize)]
#[prompt(template = "USER PROFILE:\nName: {{name}}\nRole: {{role}}")]
struct UserProfile {
    name: &'static str,
    role: &'static str,
}

let user = UserProfile {
    name: "Yui",
    role: "World-Class Pro Engineer",
};

let p = user.to_prompt();
// The following would be printed:
// USER PROFILE:
// Name: Yui
// Role: World-Class Pro Engineer

Future Directions

Image Handling Abstraction

A planned feature is to introduce a unified interface for handling image inputs across different LLM providers. This would abstract away the complexities of dealing with various data formats (e.g., Base64, URLs, local file paths) and model-specific requirements, providing a simple and consistent API for multimodal applications.