Skip to main content

Crate tpt_gpu_model_optimizer

Crate tpt_gpu_model_optimizer 

Source
Expand description

TPT Model Optimizer — hardware-aware LLM compression pipeline.

§Pipeline

GGUF source
  → HardwareProfiler   (benchmark GPU; cache to ~/.tpt/hardware_profile.json)
  → LayerSensitivityMap (rank layers by quantization sensitivity)
  → DomainMapper        (identify neurons by domain via activation analysis)
  → SurgicalPruner      (zero unwanted domain neurons)
  → MixedPrecisionAllocator (5% loss frontier per layer)
  → KvCacheCalculator   (recommend context window for remaining VRAM)
  → TptfFormat writer   (output .tptf — self-contained: weights + tokenizer + chat template)

Export: GgufExporter, Exl2Exporter
Quality: QualityBenchmark (perplexity + task accuracy)
Large models: StreamingLoader (70B+ via layer-by-layer mmap)

Re-exports§

pub use activation_capture::load_domain_maps_from_dir;
pub use activation_capture::ActivationCallback;
pub use activation_capture::ActivationCapture;
pub use activation_capture::ActivationMap;
pub use activation_capture::LayerActivations;
pub use benchmark::QualityBenchmark;
pub use calibration::CalibrationGenerator;
pub use domain_mapper::DomainMapper;
pub use export::detect::ModelFormat;
pub use import::gguf::GgufImporter;
pub use import::gguf::GgufModel;
pub use kv_calculator::KvCacheCalculator;
pub use profiler::HardwareProfiler;
pub use pruner::SurgicalPruner;
pub use quant_allocator::MixedPrecisionAllocator;
pub use sensitivity::LayerSensitivityMap;
pub use sensitivity::SensitivityConfig;
pub use streaming::StreamingLoader;
pub use tptf_format::TensorBlock;
pub use tptf_format::TptfHeader;
pub use tptf_format::TptfWriter;

Modules§

activation_capture
Activation Capture — hooks for capturing intermediate activations during inference.
benchmark
Quality Benchmark — measures perplexity and task accuracy before/after optimization.
calibration
Calibration Generator — produces hard domain-specific prompts for quality evaluation.
domain_mapper
Domain Knowledge Mapper — identifies which neurons handle specific domains.
export
import
GGUF → TPTF import pipeline.
kv_calculator
KV Cache Calculator — computes maximum context window from remaining VRAM.
profiler
Hardware profiler — benchmarks the target GPU and caches results.
pruner
Surgical Pruner — zeros FFN neurons associated with unwanted domains.
quant_allocator
Mixed-Precision Allocator — finds minimum bits per layer within quality budget.
sensitivity
Layer sensitivity map — fast pre-pass to rank layers by quantization sensitivity.
streaming
Streaming Loader — processes 70B+ models layer-by-layer via mmap.
tptf_format
TPTF file format — TPT’s self-contained model format.

Structs§

OptimizationResult
Summary produced after a full optimization run.
OptimizerConfig
Configuration for a full optimization run.