Expand description
A Rust re-implementation of transformers.AutoModelForCausalLM built on Candle.
It is model-only by design: it loads a Hugging Face-style checkpoint
directory containing config.json and .safetensors files and returns a
CausalLM trait object. Tokenization, downloading, and generation loops
live in the caller (see the examples/ directory for pairing with the
tokenizers crate).
§Quick start
use auto_causal_lm::AutoModelForCausalLM;
use candle_core::{DType, Device};
let mut model = AutoModelForCausalLM::from_pretrained_with(
"/path/to/checkpoint",
Device::Cpu,
DType::F32,
)?;§Supported architectures
- Llama / Mistral / Mixtral / Yi
- Qwen2 / Qwen3
- Phi3
- StableLM
- StarCoder2
- Granite / GraniteMoE
- OLMo
- Gemma / Gemma2 / Gemma3
- Falcon
- Bloom
- GPT-2
- Mamba / Mamba2
- RWKV6
- ChatGLM / GLM4
- DeepSeek-V2 / DeepSeek-V3
Re-exports§
pub use cache::Cache;pub use cache::CacheLayout;pub use cache::KV;pub use error::Error;pub use error::Result;pub use mapper::KVMapper;pub use mapper::MapperRegistry;pub use mapper::PassthroughMapper;pub use model::CausalLM;pub use model::CausalLMCacheExt;pub use model::KVCache;pub use model::ModelConfig;
Modules§
Structs§
- Auto
Model ForCausalLM - Entry point for loading causal language models from a local checkpoint directory.