trustformers-core 0.2.1

Core traits and utilities for TrustformeRS
Documentation

trustformers-core

Version Status SLoC Date

Core infrastructure crate providing fundamental abstractions and utilities for the TrustformeRS ecosystem.

Current State

Version 0.2.1 — last verified 2026-08-24

This crate is stable and production-ready for its real (CPU/CUDA/Metal) compute paths, serving as the foundation for all other TrustformeRS components. It provides high-performance tensor operations, layer implementations, and advanced optimization techniques. The last workspace-level test run reported to this documentation pass measured 21,370 passed / 41 skipped / 0 failed across the whole workspace (default features, 2026-08-26) — this crate's own share was not re-measured separately this pass; see TODO.md for the crate's detailed, dated status (including which GPU backends are real vs. feature-flag facades).

Features

Tensor Operations

  • Comprehensive tensor abstraction supporting multiple backends
  • SciRS2 integration for SIMD-optimized operations
  • GPU support: each backend below is an opt-in Cargo feature (no GPU backend is enabled by default, default = ["linalg"]), but the features are not equally real. Real, hardware-verified on this project's own machines: CUDA (cuda, via the Pure-Rust oxicuda), Metal (metal, via oxicuda-metal). Real code, not hardware-verified in this environment: Vulkan (vulkan, via vulkano — device/pipeline setup is real, several compute operations still return a structured "not implemented" error), ROCm (rocm — real dlopen-based HIP calls if a real AMD runtime is present on the host). Empty facades — the feature exists, the backend does not: oneapi, xla, riscv (every operation returns a structured "no runtime linked" error or runs a scalar-CPU fallback; see TODO.md for specifics). wgpu_backend/opencl exist as features but were not characterized by this documentation pass.
  • Automatic differentiation with reverse-mode and forward-mode autodiff
  • Memory-efficient operations with zero-copy views

Layer Implementations

  • Core Layers: Linear, Embedding, LayerNorm, Dropout, RMSNorm
  • Attention Mechanisms:
    • Multi-head attention (MHA) with causal masking
    • FlashAttention and FlashAttention-2 for memory efficiency
    • Multi-Query Attention (MQA) and Grouped-Query Attention (GQA)
    • PagedAttention for KV cache management
    • Optimized SDPA kernels with adaptive strategies
  • Advanced Layers: FeedForward (SwiGLU, GeGLU), PositionalEncoding, RoPE, ALiBi

Performance Optimizations

  • SIMD Operations: Optimized LayerNorm, Softmax, and RoPE implementations
  • Quantization Support: INT8, INT4, FP16, FP8, GPTQ, AWQ, GGUF/K-quants with calibration
  • Custom Kernels: Fused operations for reduced memory bandwidth
  • Kernel Tuning: Automatic hardware-aware kernel parameter optimization
  • Memory Management: Adaptive pooling with LRU, LFU, ARC, and Hybrid eviction policies
  • Conv2D: Full im2col+matmul implementation with groups and dilation
  • GPU Attention: Scaled dot-product and flash attention (tiled online-softmax); the CUDA backend (cuda feature) additionally runs a fully GPU-resident attention pipeline — QKV split, RoPE, causal softmax, KV-cache append — and batched/broadcasting matmul, avoiding host round-trips during prefill and decode

Export and Interoperability

  • ONNX Export: Complete graph construction and runtime support
  • GGML/GGUF: Advanced quantization formats including K-quants for edge deployment
  • CoreML: iOS/macOS deployment support

Advanced Features

  • Evaluation Framework: GLUE, SuperGLUE, MMLU, HellaSwag, HumanEval benchmarks
  • Monitoring: TensorBoard integration, gradient flow analysis, activation statistics
  • Caching System: Multiple eviction policies (LRU, LFU, ARC, Hybrid)
  • A/B Testing: Infrastructure for model comparison
  • Model Compression: Pruning and distillation support
  • Plugin System: Extensible architecture for custom kernels and layers
  • Tensor Debugger: Interactive watchpoints, NaN/Inf detection, operation tracing

Distributed and Parallel Computing

  • Tensor Parallelism: Column/row parallel linear layers
  • Pipeline Parallelism: Stage-based model partitioning
  • Data Parallelism: Multi-GPU training infrastructure
  • Communication Backends: NCCL, MPI, Gloo support
  • Process Groups: All-reduce, broadcast, all-gather, reduce-scatter operations

PEFT (Parameter-Efficient Fine-Tuning)

  • LoRA: Low-rank adaptation with weight merging
  • QLoRA: Quantized LoRA for memory efficiency
  • Adapters: Bottleneck adapter layers
  • Prefix Tuning: Trainable prefix embeddings
  • Prompt Tuning: Virtual token optimization

Architecture

trustformers-core/
├── src/
│   ├── tensor/             # Tensor abstractions and operations
│   ├── layers/             # Neural network layers
│   ├── attention/          # Attention mechanisms
│   ├── quantization/       # Quantization infrastructure (largest module by API surface)
│   ├── export/             # Model export formats (ONNX, GGUF, CoreML)
│   ├── kernels/            # Custom fused and SIMD compute kernels
│   ├── hardware/           # Hardware acceleration abstraction (CUDA, Metal, Vulkan, ROCm, ...)
│   ├── compiler/           # JIT compilation, kernel fusion, graph optimization
│   ├── performance/        # Benchmarking, profiling, and optimization advisor
│   ├── monitoring/         # Profiling and analysis
│   ├── parallel/           # Distributed computing
│   ├── evaluation/         # Benchmark implementations
│   └── peft.rs             # Parameter-efficient fine-tuning

Usage Example

use trustformers_core::layers::{Linear, LayerNorm, MultiHeadAttention};
use trustformers_core::tensor::Tensor;
use trustformers_core::traits::Layer;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create layers for a transformer block
    let attention = MultiHeadAttention::new(768, 12, 0.1, true)?;
    let norm1 = LayerNorm::new(vec![768], 1e-5)?;
    let ffn = Linear::new(768, 3072, true);
    let norm2 = LayerNorm::new(vec![768], 1e-5)?;

    // Run the attention sub-layer's forward pass
    let input = Tensor::randn(&[2, 128, 768])?;
    let attended = attention.forward(input)?;

    Ok(())
}

Performance

  • FlashAttention: O(N) memory complexity vs O(N²) standard
  • Quantization: 50-75% memory reduction with INT8/INT4
  • SIMD: 2-3x speedup on supported operations
  • PagedAttention: Eliminates KV cache fragmentation

Testing

The crate includes comprehensive test coverage. Workspace-wide (default features), cargo nextest run --workspace most recently measured 21,370 passed / 41 skipped / 0 failed (2026-08-26; see root README.md/TODO.md for the current figure, as this grows over time and this crate's own share was not separately re-measured this pass):

  • Property-based testing with proptest
  • Memory leak detection
  • Performance benchmarks
  • Cross-backend compatibility tests
  • Numerical stability tests with adaptive tolerance

Dependencies

  • scirs2-core: SIMD operations and parallelism
  • half: FP16/BF16 support
  • rayon: Parallel iteration (via SciRS2)
  • oxicuda-{blas,dnn,memory,driver} (optional, cuda feature): CUDA GEMM, resident attention, and refcounted GPU buffer management
  • Various serialization and utility crates

Public API

The crate exposes ~4,533 public API items (structs, enums, traits, and fns across the public API, including impl blocks — broader than a prior doc revision's narrower top-level-only count) covering tensors, layers, attention, quantization, export, evaluation, monitoring, distributed computing, PEFT, kernel tuning, and memory management.

License

Apache-2.0