litertlm 0.2.1

Safe, idiomatic Rust bindings to Google's LiteRT-LM on-device LLM engine.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# litertlm

Safe Rust bindings to Google's LiteRT-LM — on-device LLM inference.

```toml
[dependencies]
litertlm = "0.2"
```

```rust
use litertlm::{Backend, Engine, EngineSettings, SamplerParams};

let engine = Engine::new(
    EngineSettings::new("model.litertlm")
        .backend(Backend::Gpu)
        .max_num_tokens(512),
)?;

let mut session = engine.create_session(
    SamplerParams::default().top_k(40).temperature(0.7),
)?;

let response = session.generate("Explain Rust lifetimes")?;
println!("{response}");
# Ok::<(), litert_lm::Error>(())
```

See the [workspace README](https://github.com/offbit-ai/LiteRT#readme) for
the full story.

## License

Apache-2.0; see [NOTICE](https://github.com/offbit-ai/LiteRT/blob/main/NOTICE).