Expand description
Safe, idiomatic Rust bindings for LiteRT-LM.
This crate wraps the stable LiteRT-LM C API (c/engine.h), which is the supported
FFI surface for the C++ Conversation / Engine runtime documented in litert.md.
§Features
- Conversation API — chat-style
send_message/ streaming - Multimodal — vision & audio backends, image/audio content parts (path or blob)
- MTP — Multi-Token Prediction via speculative decoding
- Session API — lower-level prefill / decode / generate_content
- Tools, thinking/reasoning, constrained decoding, LoRA, benchmarks
§Quick start (text)
use litert_lm_rust::{Backend, Engine, Message};
let engine = Engine::builder("model.litertlm")
.backend(Backend::Cpu)
.build()?;
let mut conversation = engine.create_conversation(Default::default())?;
let reply = conversation.send_message(Message::user("Hello!"))?;
println!("{}", reply);§Multimodal
use litert_lm_rust::{Backend, ContentPart, Engine, Message};
let engine = Engine::builder("gemma-3n.litertlm")
.backend(Backend::Cpu)
.vision_backend(Backend::Gpu)
.audio_backend(Backend::Cpu)
.build()?;
let mut conversation = engine.create_conversation(Default::default())?;
let reply = conversation.send_message(Message::user_parts([
ContentPart::text("Describe this image:"),
ContentPart::image_path("photo.jpg"),
])?)?;§MTP (Multi-Token Prediction)
use litert_lm_rust::{Backend, Engine};
let engine = Engine::builder("model.litertlm")
.backend(Backend::Gpu)
.multi_token_prediction(true) // enable_speculative_decoding
.build()?;§Native library
Link against a built LiteRT-LM C shared library (bazel build //c:engine or equivalent).
Set LITERT_LM_LIB_DIR to the directory containing the library.
Structs§
- Benchmark
Info - Timing and throughput metrics from a session or conversation.
- Conversation
- Stateful chat conversation (manages Session + prompt template + multimodal preprocessing).
- Conversation
Config - Configuration for creating a
Conversation. - Conversation
Optional Args - Per-turn optional arguments for
Conversation::send_message. - Engine
- Heavyweight model holder. Create lightweight
Conversation/Sessionfrom it. - Engine
Builder - Builder for
Engine. - Message
- A chat message exchanged with
crate::Conversation. - NoRepeat
Ngram Config - No-repeat n-gram configuration.
- Repetition
Penalty Config - Repetition penalty configuration.
- Responses
- Model responses from a blocking generate / decode call.
- Sampler
Params - Sampling hyperparameters.
- Send
Options - Options passed alongside a message.
- Session
- Stateful generation session owned by an
Engine. - Session
Config - Configuration for a
Session. - Stream
Chunk - A single streamed chunk from LiteRT-LM.
- Stream
Event Receiver - Receiver half of a streaming generation call.
- Suppress
Tokens Config - Suppress-tokens configuration.
- Thinking
Config - Thinking / reasoning configuration.
Enums§
- Activation
Data Type - Activation data type for the engine.
- Backend
- Hardware backend for model execution.
- Constraint
Type - Constraint type for constrained decoding.
- Content
Part - A multimodal content part inside a message.
- Error
- Errors returned by the LiteRT-LM Rust bindings.
- Input
Data - Multimodal input for
crate::Session::generate_content. - LogSeverity
- Native library log severity.
- Role
- Role of a conversation participant.
- Sampler
Type - Sampler algorithm.
- Stream
Event - Events produced by async generation.
Constants§
- VERSION
- Crate version string.
Functions§
- set_
min_ log_ level - Set the minimum log severity for the native LiteRT-LM library.
Type Aliases§
- Result
- Result alias for this crate.