llm-bench 0.4.0

Compare LLM providers on your own prompts: OpenAI, Anthropic, OpenRouter, Groq, Together, Fireworks, DeepSeek, Gemini, Ollama or any OpenAI-compatible URL; time to first token, p50/p95 latency, tokens per second, error rate and cost, as a table, Markdown or JSON
Documentation