Expand description
Batch processing for high-throughput inference
Provides static and dynamic batching to maximize GPU/CPU utilization and achieve 3-10x throughput improvements.
Structs§
- Batch
Config - Configuration for batch processor
- Batch
Processor - Batch processor for high-throughput inference
- Batch
Request - A single inference request
- Batch
Stats - Statistics about batch processing
Enums§
- Batch
Strategy - Batch processing strategy