Skip to main content

Crate litert_lm_rust

Crate litert_lm_rust 

Source
Expand description

Safe, idiomatic Rust bindings for LiteRT-LM.

This crate wraps the stable LiteRT-LM C API (c/engine.h), which is the supported FFI surface for the C++ Conversation / Engine runtime documented in litert.md.

§Features

  • Conversation API — chat-style send_message / streaming
  • Multimodal — vision & audio backends, image/audio content parts (path or blob)
  • MTP — Multi-Token Prediction via speculative decoding
  • Session API — lower-level prefill / decode / generate_content
  • Tools, thinking/reasoning, constrained decoding, LoRA, benchmarks

§Quick start (text)

use litert_lm_rust::{Backend, Engine, Message};

let engine = Engine::builder("model.litertlm")
    .backend(Backend::Cpu)
    .build()?;
let mut conversation = engine.create_conversation(Default::default())?;
let reply = conversation.send_message(Message::user("Hello!"))?;
println!("{}", reply);

§Multimodal

use litert_lm_rust::{Backend, ContentPart, Engine, Message};

let engine = Engine::builder("gemma-3n.litertlm")
    .backend(Backend::Cpu)
    .vision_backend(Backend::Gpu)
    .audio_backend(Backend::Cpu)
    .build()?;
let mut conversation = engine.create_conversation(Default::default())?;
let reply = conversation.send_message(Message::user_parts([
    ContentPart::text("Describe this image:"),
    ContentPart::image_path("photo.jpg"),
])?)?;

§MTP (Multi-Token Prediction)

use litert_lm_rust::{Backend, Engine};

let engine = Engine::builder("model.litertlm")
    .backend(Backend::Gpu)
    .multi_token_prediction(true) // enable_speculative_decoding
    .build()?;

§Native library

Link against a built LiteRT-LM C shared library (bazel build //c:engine or equivalent). Set LITERT_LM_LIB_DIR to the directory containing the library.

Structs§

BenchmarkInfo
Timing and throughput metrics from a session or conversation.
Conversation
Stateful chat conversation (manages Session + prompt template + multimodal preprocessing).
ConversationConfig
Configuration for creating a Conversation.
ConversationOptionalArgs
Per-turn optional arguments for Conversation::send_message.
Engine
Heavyweight model holder. Create lightweight Conversation / Session from it.
EngineBuilder
Builder for Engine.
Message
A chat message exchanged with crate::Conversation.
NoRepeatNgramConfig
No-repeat n-gram configuration.
RepetitionPenaltyConfig
Repetition penalty configuration.
Responses
Model responses from a blocking generate / decode call.
SamplerParams
Sampling hyperparameters.
SendOptions
Options passed alongside a message.
Session
Stateful generation session owned by an Engine.
SessionConfig
Configuration for a Session.
StreamChunk
A single streamed chunk from LiteRT-LM.
StreamEventReceiver
Receiver half of a streaming generation call.
SuppressTokensConfig
Suppress-tokens configuration.
ThinkingConfig
Thinking / reasoning configuration.

Enums§

ActivationDataType
Activation data type for the engine.
Backend
Hardware backend for model execution.
ConstraintType
Constraint type for constrained decoding.
ContentPart
A multimodal content part inside a message.
Error
Errors returned by the LiteRT-LM Rust bindings.
InputData
Multimodal input for crate::Session::generate_content.
LogSeverity
Native library log severity.
Role
Role of a conversation participant.
SamplerType
Sampler algorithm.
StreamEvent
Events produced by async generation.

Constants§

VERSION
Crate version string.

Functions§

set_min_log_level
Set the minimum log severity for the native LiteRT-LM library.

Type Aliases§

Result
Result alias for this crate.