Crate runtime 

UniLLM Runtime

High-performance inference runtime for large language models.

This crate provides a clean, solid abstraction system for LLM inference with support for multiple model architectures and deployment targets.

Re-exports

pub use tensor_core::Tensor;

pub use tensor_core::Device;

pub use tensor_core::DataType;

pub use model_core::Model;

pub use model_core::ModelInputs;

pub use model_core::ModelOutputs;

pub use model_core::GenerationConfig;

pub use model_core::MemoryRequirements;

pub use model_core::ModelWeights;

pub use weight_loader_core::WeightLoader;

pub use kv_cache::KVCache;

pub use kv_cache::LayerKVCache;

pub use precompute::RoPECache;

pub use precompute::CausalMaskCache;

pub use precompute::SlidingWindowMaskCache;

Modules

benchmark

Benchmark comparison module Benchmark comparison module for UniLLM vs llama.cpp

inference

Basic inference implementation Complete inference pipeline for text generation

kv_cache

KV cache for efficient autoregressive generation KV Cache for efficient autoregressive generation

model_core

Model trait and configuration system Core Model Abstraction

models_v2

Clean model implementations using solid abstractions Models V2 - Clean implementations using solid abstractions

ollama

Ollama registry client Ollama Registry Client

precompute

Precomputed static tensors for performance (RoPE, causal masks) Precomputed Static Tensors for Performance

sampler

Sampling and decoding Greedy sampler implementation

simple_observability

Simple observability Simple High-Performance Observability

tensor_core

Unified tensor operations and device management Core Tensor Abstraction

tokenizer

Tokenization utilities Tokenizer implementation using HuggingFace tokenizers

types

Type definitions Core types for UniLLM runtime

weight_loader_core

Weight loading from various formats Core Weight Loading Abstraction

Macros

model_config

Helper macro for creating model configurations

Structs

Runtime

Main runtime instance