llmproxy

llmproxy is a lightweight server that sits between your AI tools and the actual AI providers that generate responses. Many tools expect to talk to a local AI runtime like Ollama, OpenAI's API, or llama.cpp. llmproxy pretends to be one of those local runtimes, receives the request, and forwards it to a real upstream provider such as NVIDIA, Anthropic, Google Gemini, Azure OpenAI, or any OpenAI-compatible endpoint. The benefit is that your existing tools keep working without any changes to their code. You just point them at llmproxy instead of the real local runtime.

It supports chat, completions, and embeddings, along with streaming responses. When a request fails because of a temporary upstream error, llmproxy can retry automatically. It also offers an optional response cache with configurable size and expiration, which speeds up repeated non-streaming replies. A live stats dashboard at the /stats endpoint shows metrics and process information. Inbound authentication is optional if you need to restrict who can call the proxy.

Providers are declared in a configuration file called providers.toml. When you have a single provider configured, model names stay unchanged. When you configure two or more, model names get a prefix like provider:model so you can tell them apart. You can also set a custom alias for any model. If no configuration file exists, llmproxy can fall back to reading NVIDIA environment variables, so existing setups keep working with no extra configuration. A migration command helps generate a starting file from your current environment.

The quick start involves setting your NVIDIA API key in an environment file, then running the project with Docker Compose or a prebuilt Docker image. A test request to the root URL returns a familiar message confirming the proxy is running. The project is written in Python using Flask and is released under the MIT license. Documentation covers installation, configuration, deployment, troubleshooting, and usage examples.