Expand description
NVIDIA NIM Provider - Enterprise GPU Inference
NVIDIA NIM (NVIDIA Inference Microservices) provides optimized inference for LLMs on NVIDIA GPUs. Supports both self-hosted and cloud deployments.
Setup:
- Get API key from: https://build.nvidia.com
- Configure rco:
rco config set RCO_AI_PROVIDER=nvidia RCO_API_KEY=<key> RCO_MODEL=meta/llama-3.1-8b-instruct
Docs: https://docs.nvidia.com/nim/
Structsยง
- Nvidia
Provider - Nvidia
Provider Builder - ProviderBuilder for NVIDIA NIM