async-llm 0.8.0

Async Rust client for LLM APIs - Anthropic Messages API today; fork of async-anthropic with thinking-block and prompt-cache support.
Documentation

[!NOTE] Originally the client was forked from anthropic-sdk which no longer seems to be maintained. There might still be some references, even though the code has been rewritten from scratch.

async-llm

A fork of async-anthropic, published as async-llm, adding thinking-block support, prompt caching (cache_control + cache usage fields), and assorted fixes.

A client for the anthropic messages api, written in Rust. There are plenty of clients on crates.io, but we figured we needed another one. Specifically, a straightforward builder api, robust error handling, and room to grow. Tests are also nice.

Features

  • Messages API
  • Models API
  • Tool use
  • Support all API parameters
  • Automatic backoff
  • Tracing
  • Streaming
  • Non-text messages

Installation

Add the project with cargo:

cargo add async-llm

Usage

Basic Usage

For non-streaming responses, you can use the SDK as follows:

    let client = Client::default();

    let request = CreateMessagesRequestBuilder::default()
        .model("claude-3-5-sonnet-20241022")
        .messages(vec![MessageBuilder::default()
            .role(MessageRole::User)
            .content("Hello claude!!")
            .build()
            .unwrap()])
        .build()
        .unwrap();

    let response = client.messages().create(request).await?;

    println!("{:?}", response);

See /examples for more examples.

Contributing

Contributions are welcome! This project was quickly drafted together to add anthropic support to other bosun projects, and several features are missing. If you'd like to contribute, please open an issue or a pull request.