Module llm 

[route.action] type = "llm": the LLM proxy from the config file (18.9.0).

[[route]]
  [route.match]
  path = "/v1/*"

  [route.action]
  type = "llm"

  [route.action.llm]
  audit_log          = true
  redis              = "127.0.0.1:6379"   # share budgets between replicas
  redis_password_env = "REDIS_PASSWORD"
  redis_prefix       = "rws:llm:"
  connect_timeout_ms = 10000
  read_timeout_ms    = 300000

  [[route.action.llm.provider]]         # tried in this order, per API
  name    = "openai"
  api     = "openai"                    # or "anthropic"
  url     = "https://api.openai.com"
  key_env = "OPENAI_API_KEY"

  [[route.action.llm.key]]              # a virtual key clients send
  identity      = "alice"
  key_env       = "ALICE_LLM_KEY"
  budget_tokens = 2000000               # optional, with window_secs
  window_secs   = 86400

Secrets never sit in the file: provider keys, virtual keys and the Redis password are read from the environment variables named.

Structs

LlmConfig

LlmKeyConfig

LlmProviderConfig