Skip to main content

Crate auto_causal_lm

Crate auto_causal_lm 

Source
Expand description

A Rust re-implementation of transformers.AutoModelForCausalLM built on Candle.

It is model-only by design: it loads a Hugging Face-style checkpoint directory containing config.json and .safetensors files and returns a CausalLM trait object. Tokenization, downloading, and generation loops live in the caller (see the examples/ directory for pairing with the tokenizers crate).

§Quick start

use auto_causal_lm::AutoModelForCausalLM;
use candle_core::{DType, Device};

let mut model = AutoModelForCausalLM::from_pretrained_with(
    "/path/to/checkpoint",
    Device::Cpu,
    DType::F32,
)?;

§Supported architectures

  • Llama / Mistral / Mixtral / Yi
  • Qwen2 / Qwen3
  • Phi3
  • StableLM
  • StarCoder2
  • Granite / GraniteMoE
  • OLMo
  • Gemma / Gemma2 / Gemma3
  • Falcon
  • Bloom
  • GPT-2
  • Mamba / Mamba2
  • RWKV6
  • ChatGLM / GLM4
  • DeepSeek-V2 / DeepSeek-V3

Re-exports§

pub use cache::Cache;
pub use cache::CacheLayout;
pub use cache::KV;
pub use error::Error;
pub use error::Result;
pub use mapper::KVMapper;
pub use mapper::MapperRegistry;
pub use mapper::PassthroughMapper;
pub use model::CausalLM;
pub use model::CausalLMCacheExt;
pub use model::KVCache;
pub use model::ModelConfig;

Modules§

cache
common
error
families
mapper
model
registry

Structs§

AutoModelForCausalLM
Entry point for loading causal language models from a local checkpoint directory.